From 2329cce01a315669393ed11a0ac335c2f257e8cc Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 14 Nov 2024 22:22:05 +0000 Subject: [PATCH] ** deploy nuprl/website@5e7275f02a8cea9a4a51f769b025327137c8fec2 --- blog/.frog/build | 1 + .../04/29/welcome-to-the-prl-blog/index.html | 190 + .../index.html | 183 + .../index.html | 237 + .../index.html | 609 ++ .../index.html | 194 + .../index.html | 183 + .../27/tutorial-using-racket-s-ffi/index.html | 669 +++ .../29/tutorial-racket-ffi-part-2/index.html | 880 +++ .../11/tutorial-racket-ffi-part-3/index.html | 1127 ++++ .../index.html | 1247 ++++ .../08/03/a-few-cores-too-many/index.html | 215 + .../index.html | 199 + blog/2016/10/11/compcert-overview/index.html | 198 + .../index.html | 223 + blog/2016/10/19/history-of-actors/index.html | 172 + .../10/31/meaningful-distinctions/index.html | 226 + .../11/02/beta-reduction-part-1/index.html | 459 ++ .../index.html | 238 + blog/2016/11/17/src-submissions/index.html | 212 + .../index.html | 169 + .../index.html | 232 + .../index.html | 199 + .../index.html | 169 + .../index.html | 197 + .../02/15/introducing-hopl-2017/index.html | 214 + .../index.html | 291 + .../21/datalog-for-static-analysis/index.html | 204 + .../index.html | 193 + .../pliss-oregon-without-the-greek/index.html | 198 + blog/2017/03/03/plt-redex-mf-apply/index.html | 515 ++ .../index.html | 187 + .../index.html | 187 + .../index.html | 169 + .../index.html | 227 + .../index.html | 192 + blog/2017/04/20/refinement-types/index.html | 184 + blog/2017/04/25/prl-at-snapl-17/index.html | 206 + .../2017/04/28/what-is-soft-typing/index.html | 268 + .../index.html | 175 + blog/2017/05/04/rank-polymorphism/index.html | 207 + .../index.html | 185 + .../05/15/artifacts-for-semantics/index.html | 210 + .../index.html | 574 ++ .../index.html | 207 + .../index.html | 190 + blog/2017/06/05/report-pliss-2017/index.html | 223 + .../index.html | 440 ++ .../index.html | 169 + .../index.html | 229 + .../index.html | 1042 ++++ blog/2017/07/17/continuations/index.html | 178 + blog/2017/07/19/trees-1973/index.html | 180 + .../index.html | 211 + .../index.html | 230 + .../closure-conversion-as-coyoneda/index.html | 317 + .../index.html | 169 + blog/2017/09/25/plt-redex-faq/index.html | 649 ++ .../index.html | 438 ++ .../index.html | 297 + .../index.html | 169 + .../index.html | 169 + .../index.html | 169 + .../index.html | 169 + .../index.html | 192 + .../index.html | 284 + .../index.html | 226 + .../index.html | 886 +++ blog/2018/11/24/disappearing-code/index.html | 217 + .../11/30/turnstile-mailing-list/index.html | 173 + .../02/java-and-migratory-typing/index.html | 456 ++ .../index.html | 236 + .../01/28/on-stack-replacement/index.html | 184 + .../writing-a-paper-with-scribble/index.html | 495 ++ .../index.html | 198 + .../index.html | 339 ++ .../index.html | 169 + .../05/lexical-and-dynamic-scope/index.html | 319 + .../10/four-kinds-of-scoping-in-r/index.html | 488 ++ blog/2019/09/10/scoping-in-r/index.html | 674 +++ .../index.html | 258 + .../prl-offsite-2019-retrospective/index.html | 284 + .../index.html | 577 ++ .../index.html | 493 ++ .../index.html | 516 ++ .../12/23/deep-and-shallow-types/index.html | 186 + .../index.html | 405 ++ .../index.html" | 167 + blog/feeds/1st-blog-post.atom.xml | 39 + blog/feeds/1st-blog-post.rss.xml | 39 + blog/feeds/Author-Aaron-Weiss.atom.xml | 16 + blog/feeds/Author-Aaron-Weiss.rss.xml | 16 + blog/feeds/Author-Alexi-Turcotte.atom.xml | 45 + blog/feeds/Author-Alexi-Turcotte.rss.xml | 45 + blog/feeds/Author-Artem-Pelenitsyn.atom.xml | 131 + blog/feeds/Author-Artem-Pelenitsyn.rss.xml | 121 + blog/feeds/Author-Asumu-Takikawa.atom.xml | 2280 +++++++ blog/feeds/Author-Asumu-Takikawa.rss.xml | 2274 +++++++ blog/feeds/Author-Ben-Chung.atom.xml | 152 + blog/feeds/Author-Ben-Chung.rss.xml | 136 + blog/feeds/Author-Ben-Greenman.atom.xml | 4425 ++++++++++++++ blog/feeds/Author-Ben-Greenman.rss.xml | 4381 ++++++++++++++ blog/feeds/Author-Daniel-Patterson.atom.xml | 108 + blog/feeds/Author-Daniel-Patterson.rss.xml | 102 + blog/feeds/Author-Dustin-Jamner.atom.xml | 84 + blog/feeds/Author-Dustin-Jamner.rss.xml | 76 + blog/feeds/Author-Gabriel-Scherer.atom.xml | 1233 ++++ blog/feeds/Author-Gabriel-Scherer.rss.xml | 1211 ++++ blog/feeds/Author-Jan-Vitek.atom.xml | 45 + blog/feeds/Author-Jan-Vitek.rss.xml | 45 + blog/feeds/Author-Jonathan-Schuster.atom.xml | 16 + blog/feeds/Author-Jonathan-Schuster.rss.xml | 16 + blog/feeds/Author-Julia-Belyakova.atom.xml | 84 + blog/feeds/Author-Julia-Belyakova.rss.xml | 76 + blog/feeds/Author-Justin-Slepak.atom.xml | 54 + blog/feeds/Author-Justin-Slepak.rss.xml | 54 + blog/feeds/Author-Kevin-Clancy.atom.xml | 168 + blog/feeds/Author-Kevin-Clancy.rss.xml | 166 + blog/feeds/Author-Leif-Andersen.atom.xml | 325 + blog/feeds/Author-Leif-Andersen.rss.xml | 317 + blog/feeds/Author-Li-Yao-Xia.atom.xml | 289 + blog/feeds/Author-Li-Yao-Xia.rss.xml | 287 + blog/feeds/Author-Matt-Kolosick.atom.xml | 84 + blog/feeds/Author-Matt-Kolosick.rss.xml | 76 + blog/feeds/Author-Max-New.atom.xml | 535 ++ blog/feeds/Author-Max-New.rss.xml | 529 ++ blog/feeds/Author-Milo-Davis.atom.xml | 426 ++ blog/feeds/Author-Milo-Davis.rss.xml | 410 ++ blog/feeds/Author-Ming-Ho-Yee.atom.xml | 1242 ++++ blog/feeds/Author-Ming-Ho-Yee.rss.xml | 1216 ++++ blog/feeds/Author-Olek-Gierczak.atom.xml | 133 + blog/feeds/Author-Olek-Gierczak.rss.xml | 131 + ...Author-Olivier-Flu\314\210ckiger.atom.xml" | 16 + .../Author-Olivier-Flu\314\210ckiger.rss.xml" | 16 + blog/feeds/Author-Rob-Kleffner.atom.xml | 34 + blog/feeds/Author-Rob-Kleffner.rss.xml | 34 + blog/feeds/Author-Sam-Caldwell.atom.xml | 1344 +++++ blog/feeds/Author-Sam-Caldwell.rss.xml | 1324 ++++ blog/feeds/Author-Stephen-Chang.atom.xml | 20 + blog/feeds/Author-Stephen-Chang.rss.xml | 20 + blog/feeds/Author-Tony-Garnock-Jones.atom.xml | 65 + blog/feeds/Author-Tony-Garnock-Jones.rss.xml | 61 + blog/feeds/Author-William-J-Bowman.atom.xml | 77 + blog/feeds/Author-William-J-Bowman.rss.xml | 67 + blog/feeds/Author-Zeina-Migeed.atom.xml | 133 + blog/feeds/Author-Zeina-Migeed.rss.xml | 131 + blog/feeds/CRDTs.atom.xml | 144 + blog/feeds/CRDTs.rss.xml | 144 + blog/feeds/Emacs.atom.xml | 70 + blog/feeds/Emacs.rss.xml | 70 + blog/feeds/FFI.atom.xml | 2203 +++++++ blog/feeds/FFI.rss.xml | 2199 +++++++ blog/feeds/Gradual-Typing.atom.xml | 32 + blog/feeds/Gradual-Typing.rss.xml | 32 + blog/feeds/HOPL.atom.xml | 502 ++ blog/feeds/HOPL.rss.xml | 478 ++ blog/feeds/ICFP.atom.xml | 41 + blog/feeds/ICFP.rss.xml | 41 + blog/feeds/NEPLS.atom.xml | 69 + blog/feeds/NEPLS.rss.xml | 67 + blog/feeds/PI-meeting.atom.xml | 154 + blog/feeds/PI-meeting.rss.xml | 152 + blog/feeds/PL-Junior.atom.xml | 127 + blog/feeds/PL-Junior.rss.xml | 113 + blog/feeds/PLT-Redex.atom.xml | 853 +++ blog/feeds/PLT-Redex.rss.xml | 849 +++ blog/feeds/Racket.atom.xml | 3322 +++++++++++ blog/feeds/Racket.rss.xml | 3314 ++++++++++ "blog/feeds/R\314\214-JIT.atom.xml" | 16 + "blog/feeds/R\314\214-JIT.rss.xml" | 16 + blog/feeds/Scribble.atom.xml | 756 +++ blog/feeds/Scribble.rss.xml | 754 +++ blog/feeds/System-Administration.atom.xml | 70 + blog/feeds/System-Administration.rss.xml | 70 + blog/feeds/Takikawa-constant.atom.xml | 133 + blog/feeds/Takikawa-constant.rss.xml | 131 + blog/feeds/Yoneda.atom.xml | 164 + blog/feeds/Yoneda.rss.xml | 164 + blog/feeds/about.atom.xml | 39 + blog/feeds/about.rss.xml | 39 + blog/feeds/adjunction.atom.xml | 85 + blog/feeds/adjunction.rss.xml | 85 + blog/feeds/all.atom.xml | 3645 +++++++++++ blog/feeds/all.rss.xml | 3605 +++++++++++ blog/feeds/announcement.atom.xml | 20 + blog/feeds/announcement.rss.xml | 20 + blog/feeds/array-language.atom.xml | 54 + blog/feeds/array-language.rss.xml | 54 + blog/feeds/benchmarking.atom.xml | 62 + blog/feeds/benchmarking.rss.xml | 62 + blog/feeds/beta.atom.xml | 306 + blog/feeds/beta.rss.xml | 306 + blog/feeds/books.atom.xml | 16 + blog/feeds/books.rss.xml | 16 + blog/feeds/bugs.atom.xml | 37 + blog/feeds/bugs.rss.xml | 37 + blog/feeds/calculus.atom.xml | 306 + blog/feeds/calculus.rss.xml | 306 + blog/feeds/castle.atom.xml | 45 + blog/feeds/castle.rss.xml | 45 + blog/feeds/category-theory.atom.xml | 535 ++ blog/feeds/category-theory.rss.xml | 529 ++ blog/feeds/clojure.atom.xml | 252 + blog/feeds/clojure.rss.xml | 252 + blog/feeds/clojurescript.atom.xml | 252 + blog/feeds/clojurescript.rss.xml | 252 + blog/feeds/closure-conversion.atom.xml | 164 + blog/feeds/closure-conversion.rss.xml | 164 + blog/feeds/coYoneda.atom.xml | 164 + blog/feeds/coYoneda.rss.xml | 164 + blog/feeds/compiler-correctness.atom.xml | 45 + blog/feeds/compiler-correctness.rss.xml | 45 + blog/feeds/compilers.atom.xml | 164 + blog/feeds/compilers.rss.xml | 164 + blog/feeds/complete-monitoring.atom.xml | 105 + blog/feeds/complete-monitoring.rss.xml | 105 + blog/feeds/conference.atom.xml | 69 + blog/feeds/conference.rss.xml | 67 + blog/feeds/constructions.atom.xml | 73 + blog/feeds/constructions.rss.xml | 73 + blog/feeds/coq.atom.xml | 176 + blog/feeds/coq.rss.xml | 174 + blog/feeds/dawn-of-the-digital-era.atom.xml | 30 + blog/feeds/dawn-of-the-digital-era.rss.xml | 30 + blog/feeds/dear-diary.atom.xml | 946 +++ blog/feeds/dear-diary.rss.xml | 944 +++ blog/feeds/definitions.atom.xml | 166 + blog/feeds/definitions.rss.xml | 166 + blog/feeds/dissertation.atom.xml | 42 + blog/feeds/dissertation.rss.xml | 40 + blog/feeds/dsl.atom.xml | 733 +++ blog/feeds/dsl.rss.xml | 733 +++ blog/feeds/dynamic-typing.atom.xml | 22 + blog/feeds/dynamic-typing.rss.xml | 22 + blog/feeds/event.atom.xml | 178 + blog/feeds/event.rss.xml | 172 + blog/feeds/eventual-consistency.atom.xml | 144 + blog/feeds/eventual-consistency.rss.xml | 144 + blog/feeds/extended-abstract.atom.xml | 247 + blog/feeds/extended-abstract.rss.xml | 243 + blog/feeds/final-encoding.atom.xml | 285 + blog/feeds/final-encoding.rss.xml | 285 + blog/feeds/galois-connection.atom.xml | 85 + blog/feeds/galois-connection.rss.xml | 85 + blog/feeds/garbage-collection.atom.xml | 456 ++ blog/feeds/garbage-collection.rss.xml | 456 ++ blog/feeds/ghc.atom.xml | 456 ++ blog/feeds/ghc.rss.xml | 456 ++ blog/feeds/gradual-typing.atom.xml | 535 ++ blog/feeds/gradual-typing.rss.xml | 521 ++ blog/feeds/haskell.atom.xml | 456 ++ blog/feeds/haskell.rss.xml | 456 ++ blog/feeds/higher-order-contracts.atom.xml | 186 + blog/feeds/higher-order-contracts.rss.xml | 186 + blog/feeds/history.atom.xml | 282 + blog/feeds/history.rss.xml | 274 + blog/feeds/icfp.atom.xml | 85 + blog/feeds/icfp.rss.xml | 85 + blog/feeds/instrumentation.atom.xml | 456 ++ blog/feeds/instrumentation.rss.xml | 456 ++ blog/feeds/interactive-syntax.atom.xml | 252 + blog/feeds/interactive-syntax.rss.xml | 252 + blog/feeds/java.atom.xml | 303 + blog/feeds/java.rss.xml | 303 + blog/feeds/lambda.atom.xml | 306 + blog/feeds/lambda.rss.xml | 306 + blog/feeds/lang-extension.atom.xml | 362 ++ blog/feeds/lang-extension.rss.xml | 362 ++ blog/feeds/language-implementation.atom.xml | 83 + blog/feeds/language-implementation.rss.xml | 81 + blog/feeds/language.atom.xml | 733 +++ blog/feeds/language.rss.xml | 733 +++ blog/feeds/latency.atom.xml | 456 ++ blog/feeds/latency.rss.xml | 456 ++ blog/feeds/lectures.atom.xml | 83 + blog/feeds/lectures.rss.xml | 81 + blog/feeds/lost-time.atom.xml | 62 + blog/feeds/lost-time.rss.xml | 62 + blog/feeds/math.atom.xml | 520 ++ blog/feeds/math.rss.xml | 516 ++ blog/feeds/migratory-typing.atom.xml | 874 +++ blog/feeds/migratory-typing.rss.xml | 860 +++ blog/feeds/monotonicity.atom.xml | 144 + blog/feeds/monotonicity.rss.xml | 144 + blog/feeds/observational-equivalence.atom.xml | 285 + blog/feeds/observational-equivalence.rss.xml | 285 + blog/feeds/ocaml.atom.xml | 456 ++ blog/feeds/ocaml.rss.xml | 456 ++ blog/feeds/offsite.atom.xml | 550 ++ blog/feeds/offsite.rss.xml | 546 ++ blog/feeds/package.atom.xml | 362 ++ blog/feeds/package.rss.xml | 362 ++ blog/feeds/performance.atom.xml | 188 + blog/feeds/performance.rss.xml | 184 + blog/feeds/pliss.atom.xml | 70 + blog/feeds/pliss.rss.xml | 70 + blog/feeds/proceedings.atom.xml | 30 + blog/feeds/proceedings.rss.xml | 30 + blog/feeds/r.atom.xml | 849 +++ blog/feeds/r.rss.xml | 847 +++ blog/feeds/racket.atom.xml | 486 ++ blog/feeds/racket.rss.xml | 484 ++ blog/feeds/reduction.atom.xml | 306 + blog/feeds/reduction.rss.xml | 306 + blog/feeds/scope.atom.xml | 1008 ++++ blog/feeds/scope.rss.xml | 1004 ++++ blog/feeds/semantics.atom.xml | 315 + blog/feeds/semantics.rss.xml | 313 + blog/feeds/statistics.atom.xml | 133 + blog/feeds/statistics.rss.xml | 131 + blog/feeds/transient.atom.xml | 1409 +++++ blog/feeds/transient.rss.xml | 1403 +++++ blog/feeds/turnstile.atom.xml | 746 +++ blog/feeds/turnstile.rss.xml | 744 +++ blog/feeds/tutorial.atom.xml | 5303 +++++++++++++++++ blog/feeds/tutorial.rss.xml | 5283 ++++++++++++++++ blog/feeds/tutorials.atom.xml | 16 + blog/feeds/tutorials.rss.xml | 16 + blog/feeds/typed-racket.atom.xml | 1113 ++++ blog/feeds/typed-racket.rss.xml | 1109 ++++ blog/feeds/types.atom.xml | 144 + blog/feeds/types.rss.xml | 144 + blog/feeds/visr.atom.xml | 252 + blog/feeds/visr.rss.xml | 252 + blog/feeds/windows.atom.xml | 37 + blog/feeds/windows.rss.xml | 37 + .../pict.png | Bin 0 -> 4387 bytes .../pict.png | Bin 0 -> 7156 bytes .../pict_2.png | Bin 0 -> 4520 bytes .../pict.png | Bin 0 -> 3615 bytes .../pict_2.png | Bin 0 -> 3615 bytes blog/index-2.html | 260 + blog/index-3.html | 256 + blog/index-4.html | 272 + blog/index-5.html | 292 + blog/index-6.html | 259 + blog/index-7.html | 258 + blog/index-8.html | 285 + blog/index-9.html | 257 + blog/index.html | 289 + blog/sitemap.txt | 87 + blog/tags/1st-blog-post.html | 128 + blog/tags/Author-Aaron-Weiss.html | 125 + blog/tags/Author-Alexi-Turcotte.html | 130 + blog/tags/Author-Artem-Pelenitsyn.html | 145 + blog/tags/Author-Asumu-Takikawa.html | 202 + blog/tags/Author-Ben-Chung.html | 159 + blog/tags/Author-Ben-Greenman-2.html | 265 + blog/tags/Author-Ben-Greenman-3.html | 262 + blog/tags/Author-Ben-Greenman-4.html | 138 + blog/tags/Author-Ben-Greenman.html | 259 + blog/tags/Author-Daniel-Patterson.html | 160 + blog/tags/Author-Dustin-Jamner.html | 130 + blog/tags/Author-Gabriel-Scherer-2.html | 136 + blog/tags/Author-Gabriel-Scherer.html | 289 + blog/tags/Author-Jan-Vitek.html | 128 + blog/tags/Author-Jonathan-Schuster.html | 125 + blog/tags/Author-Julia-Belyakova.html | 130 + blog/tags/Author-Justin-Slepak.html | 140 + blog/tags/Author-Kevin-Clancy.html | 140 + blog/tags/Author-Leif-Andersen.html | 195 + blog/tags/Author-Li-Yao-Xia.html | 138 + blog/tags/Author-Matt-Kolosick.html | 130 + blog/tags/Author-Max-New.html | 174 + blog/tags/Author-Milo-Davis.html | 160 + blog/tags/Author-Ming-Ho-Yee.html | 232 + blog/tags/Author-Olek-Gierczak.html | 128 + .../Author-Olivier-Flu\314\210ckiger.html" | 125 + blog/tags/Author-Rob-Kleffner.html | 127 + blog/tags/Author-Sam-Caldwell.html | 179 + blog/tags/Author-Stephen-Chang.html | 128 + blog/tags/Author-Tony-Garnock-Jones.html | 150 + blog/tags/Author-William-J-Bowman.html | 167 + blog/tags/Author-Zeina-Migeed.html | 128 + blog/tags/CRDTs.html | 128 + blog/tags/Emacs.html | 130 + blog/tags/FFI.html | 184 + blog/tags/Gradual-Typing.html | 127 + blog/tags/HOPL-2.html | 148 + blog/tags/HOPL.html | 244 + blog/tags/ICFP.html | 128 + blog/tags/NEPLS.html | 141 + blog/tags/PI-meeting.html | 150 + blog/tags/PL-Junior.html | 147 + blog/tags/PLT-Redex.html | 143 + blog/tags/Racket.html | 212 + "blog/tags/R\314\214-JIT.html" | 125 + blog/tags/Scribble.html | 141 + blog/tags/System-Administration.html | 130 + blog/tags/Takikawa-constant.html | 128 + blog/tags/Yoneda.html | 134 + blog/tags/about.html | 128 + blog/tags/adjunction.html | 128 + blog/tags/announcement.html | 128 + blog/tags/array-language.html | 140 + blog/tags/benchmarking.html | 128 + blog/tags/beta.html | 128 + blog/tags/books.html | 125 + blog/tags/bugs.html | 137 + blog/tags/by-Aaron-Weiss.html | 10 + blog/tags/by-Alexi-Turcotte.html | 10 + blog/tags/by-Artem-Pelenitsyn.html | 10 + blog/tags/by-Asumu-Takikawa.html | 10 + blog/tags/by-Ben-Chung.html | 10 + blog/tags/by-Ben-Greenman-2.html | 10 + blog/tags/by-Ben-Greenman-3.html | 10 + blog/tags/by-Ben-Greenman-4.html | 10 + blog/tags/by-Ben-Greenman.html | 10 + blog/tags/by-Daniel-Patterson.html | 10 + blog/tags/by-Dustin-Jamner.html | 10 + blog/tags/by-Gabriel-Scherer-2.html | 10 + blog/tags/by-Gabriel-Scherer.html | 10 + blog/tags/by-Jan-Vitek.html | 10 + blog/tags/by-Jonathan-Schuster.html | 10 + blog/tags/by-Julia-Belyakova.html | 10 + blog/tags/by-Justin-Slepak.html | 10 + blog/tags/by-Kevin-Clancy.html | 10 + blog/tags/by-Leif-Andersen.html | 10 + blog/tags/by-Li-Yao-Xia.html | 10 + blog/tags/by-Matt-Kolosick.html | 10 + blog/tags/by-Max-New.html | 10 + blog/tags/by-Milo-Davis.html | 10 + blog/tags/by-Ming-Ho-Yee.html | 10 + blog/tags/by-Olek-Gierczak.html | 10 + blog/tags/by-Rob-Kleffner.html | 10 + blog/tags/by-Sam-Caldwell.html | 10 + blog/tags/by-Stephen-Chang.html | 10 + blog/tags/by-Tony-Garnock-Jones.html | 10 + blog/tags/by-William-J-Bowman.html | 10 + blog/tags/by-Zeina-Migeed.html | 10 + blog/tags/calculus.html | 128 + blog/tags/castle.html | 130 + blog/tags/category-theory.html | 174 + blog/tags/clojure.html | 151 + blog/tags/clojurescript.html | 151 + blog/tags/closure-conversion.html | 134 + blog/tags/coYoneda.html | 134 + blog/tags/compiler-correctness.html | 128 + blog/tags/compilers.html | 134 + blog/tags/complete-monitoring.html | 128 + blog/tags/conference.html | 141 + blog/tags/constructions.html | 133 + blog/tags/coq.html | 143 + blog/tags/dawn-of-the-digital-era.html | 128 + blog/tags/dear-diary.html | 141 + blog/tags/definitions.html | 130 + blog/tags/dissertation.html | 143 + blog/tags/dsl.html | 134 + blog/tags/dynamic-typing.html | 127 + blog/tags/event.html | 171 + blog/tags/eventual-consistency.html | 128 + blog/tags/extended-abstract.html | 153 + blog/tags/final-encoding.html | 130 + blog/tags/galois-connection.html | 128 + blog/tags/garbage-collection.html | 145 + blog/tags/ghc.html | 145 + blog/tags/gradual-typing.html | 213 + blog/tags/haskell.html | 145 + blog/tags/higher-order-contracts.html | 128 + blog/tags/history.html | 193 + blog/tags/icfp.html | 128 + blog/tags/instrumentation.html | 145 + blog/tags/interactive-syntax.html | 151 + blog/tags/java.html | 128 + blog/tags/lambda.html | 128 + blog/tags/lang-extension.html | 130 + blog/tags/language-implementation.html | 143 + blog/tags/language.html | 134 + blog/tags/latency.html | 145 + blog/tags/lectures.html | 143 + blog/tags/lost-time.html | 128 + blog/tags/math.html | 162 + blog/tags/migratory-typing.html | 210 + blog/tags/monotonicity.html | 128 + blog/tags/observational-equivalence.html | 130 + blog/tags/ocaml.html | 145 + blog/tags/offsite.html | 141 + blog/tags/package.html | 130 + blog/tags/performance.html | 141 + blog/tags/pliss.html | 130 + blog/tags/proceedings.html | 128 + blog/tags/r.html | 145 + blog/tags/racket.html | 167 + blog/tags/reduction.html | 128 + blog/tags/scope.html | 160 + blog/tags/semantics.html | 138 + blog/tags/statistics.html | 128 + blog/tags/transient.html | 167 + blog/tags/turnstile.html | 147 + blog/tags/tutorial.html | 280 + blog/tags/tutorials.html | 125 + blog/tags/typed-racket.html | 154 + blog/tags/types.html | 128 + blog/tags/visr.html | 151 + blog/tags/windows.html | 137 + contact.html | 120 + index.html | 58 + new-members.html | 93 + people.html | 482 ++ publications.html | 1788 ++++++ seminars.html | 1949 ++++++ software.html | 66 + teaching.html | 287 + 503 files changed, 158827 insertions(+) create mode 100644 blog/.frog/build create mode 100644 blog/2016/04/29/welcome-to-the-prl-blog/index.html create mode 100644 blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/index.html create mode 100644 blog/2016/05/18/gradual-typing-across-the-spectrum/index.html create mode 100644 blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/index.html create mode 100644 blog/2016/06/07/icfp-2016-looking-for-student-volunteers/index.html create mode 100644 blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/index.html create mode 100644 blog/2016/06/27/tutorial-using-racket-s-ffi/index.html create mode 100644 blog/2016/06/29/tutorial-racket-ffi-part-2/index.html create mode 100644 blog/2016/07/11/tutorial-racket-ffi-part-3/index.html create mode 100644 blog/2016/08/02/tutorial-zero-to-sixty-in-racket/index.html create mode 100644 blog/2016/08/03/a-few-cores-too-many/index.html create mode 100644 blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/index.html create mode 100644 blog/2016/10/11/compcert-overview/index.html create mode 100644 blog/2016/10/17/emacs-daemon-for-fast-editor-startup/index.html create mode 100644 blog/2016/10/19/history-of-actors/index.html create mode 100644 blog/2016/10/31/meaningful-distinctions/index.html create mode 100644 blog/2016/11/02/beta-reduction-part-1/index.html create mode 100644 blog/2016/11/16/understanding-constructive-galois-connections/index.html create mode 100644 blog/2016/11/17/src-submissions/index.html create mode 100644 blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/index.html create mode 100644 blog/2016/12/17/measuring-the-submission-review-balance/index.html create mode 100644 blog/2017/01/02/fall-2016-pl-junior-retrospective/index.html create mode 100644 blog/2017/01/03/-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src/index.html create mode 100644 blog/2017/02/15/conversational-context-and-concurrency/index.html create mode 100644 blog/2017/02/15/introducing-hopl-2017/index.html create mode 100644 blog/2017/02/21/bullets-are-good-for-your-coq-proofs/index.html create mode 100644 blog/2017/02/21/datalog-for-static-analysis/index.html create mode 100644 blog/2017/02/28/linear-types-for-low-level-languages/index.html create mode 100644 blog/2017/02/28/pliss-oregon-without-the-greek/index.html create mode 100644 blog/2017/03/03/plt-redex-mf-apply/index.html create mode 100644 blog/2017/03/10/type-inference-in-stack-based-programming-languages/index.html create mode 100644 blog/2017/03/15/tracing-jits-for-dynamic-languages/index.html create mode 100644 blog/2017/03/24/-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness/index.html create mode 100644 blog/2017/04/04/top-five-results-of-the-past-50-years-of-programming-languages-research/index.html create mode 100644 blog/2017/04/17/type-directed-compilation-parts-i-and-ii/index.html create mode 100644 blog/2017/04/20/refinement-types/index.html create mode 100644 blog/2017/04/25/prl-at-snapl-17/index.html create mode 100644 blog/2017/04/28/what-is-soft-typing/index.html create mode 100644 blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/index.html create mode 100644 blog/2017/05/04/rank-polymorphism/index.html create mode 100644 blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/index.html create mode 100644 blog/2017/05/15/artifacts-for-semantics/index.html create mode 100644 blog/2017/05/23/building-a-website-with-scribble/index.html create mode 100644 blog/2017/05/24/programming-language-conference-in-russia/index.html create mode 100644 blog/2017/05/26/racket-6-9-and-windows-10-creators-update/index.html create mode 100644 blog/2017/06/05/report-pliss-2017/index.html create mode 100644 blog/2017/06/05/syntactic-parametricity-strikes-again/index.html create mode 100644 blog/2017/06/09/-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html/index.html create mode 100644 blog/2017/06/16/spring-2017-pl-junior-retrospective/index.html create mode 100644 blog/2017/06/24/quotes-and-stories-from-turing-50/index.html create mode 100644 blog/2017/07/17/continuations/index.html create mode 100644 blog/2017/07/19/trees-1973/index.html create mode 100644 blog/2017/08/13/reviews-and-author-responses-we-should-stop-asking-for-500-word-responses/index.html create mode 100644 blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/index.html create mode 100644 blog/2017/08/28/closure-conversion-as-coyoneda/index.html create mode 100644 blog/2017/08/29/-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017/index.html create mode 100644 blog/2017/09/25/plt-redex-faq/index.html create mode 100644 blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/index.html create mode 100644 blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/index.html create mode 100644 blog/2018/01/17/-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html/index.html create mode 100644 blog/2018/01/19/-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist/index.html create mode 100644 blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/index.html create mode 100644 blog/2018/04/23/-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html/index.html create mode 100644 blog/2018/04/27/the-racket-school-2018-create-your-own-language/index.html create mode 100644 blog/2018/05/08/sampling-gradual-typing-performance/index.html create mode 100644 blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html create mode 100644 blog/2018/10/22/defining-local-bindings-in-turnstile-languages/index.html create mode 100644 blog/2018/11/24/disappearing-code/index.html create mode 100644 blog/2018/11/30/turnstile-mailing-list/index.html create mode 100644 blog/2018/12/02/java-and-migratory-typing/index.html create mode 100644 blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/index.html create mode 100644 blog/2019/01/28/on-stack-replacement/index.html create mode 100644 blog/2019/02/17/writing-a-paper-with-scribble/index.html create mode 100644 blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/index.html create mode 100644 blog/2019/04/07/forgetful-and-heedful-contracts/index.html create mode 100644 blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/index.html create mode 100644 blog/2019/09/05/lexical-and-dynamic-scope/index.html create mode 100644 blog/2019/09/10/four-kinds-of-scoping-in-r/index.html create mode 100644 blog/2019/09/10/scoping-in-r/index.html create mode 100644 blog/2019/10/31/complete-monitors-for-gradual-types/index.html create mode 100644 blog/2019/12/12/prl-offsite-2019-retrospective/index.html create mode 100644 blog/2020/01/15/the-typed-racket-optimizer-vs-transient/index.html create mode 100644 blog/2020/10/15/transient-answers-old-questions/index.html create mode 100644 blog/2020/11/12/transient-for-optional-and-keyword-functions/index.html create mode 100644 blog/2020/12/23/deep-and-shallow-types/index.html create mode 100644 blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/index.html create mode 100644 "blog/2022/02/16/-r\314\214-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/index.html" create mode 100644 blog/feeds/1st-blog-post.atom.xml create mode 100644 blog/feeds/1st-blog-post.rss.xml create mode 100644 blog/feeds/Author-Aaron-Weiss.atom.xml create mode 100644 blog/feeds/Author-Aaron-Weiss.rss.xml create mode 100644 blog/feeds/Author-Alexi-Turcotte.atom.xml create mode 100644 blog/feeds/Author-Alexi-Turcotte.rss.xml create mode 100644 blog/feeds/Author-Artem-Pelenitsyn.atom.xml create mode 100644 blog/feeds/Author-Artem-Pelenitsyn.rss.xml create mode 100644 blog/feeds/Author-Asumu-Takikawa.atom.xml create mode 100644 blog/feeds/Author-Asumu-Takikawa.rss.xml create mode 100644 blog/feeds/Author-Ben-Chung.atom.xml create mode 100644 blog/feeds/Author-Ben-Chung.rss.xml create mode 100644 blog/feeds/Author-Ben-Greenman.atom.xml create mode 100644 blog/feeds/Author-Ben-Greenman.rss.xml create mode 100644 blog/feeds/Author-Daniel-Patterson.atom.xml create mode 100644 blog/feeds/Author-Daniel-Patterson.rss.xml create mode 100644 blog/feeds/Author-Dustin-Jamner.atom.xml create mode 100644 blog/feeds/Author-Dustin-Jamner.rss.xml create mode 100644 blog/feeds/Author-Gabriel-Scherer.atom.xml create mode 100644 blog/feeds/Author-Gabriel-Scherer.rss.xml create mode 100644 blog/feeds/Author-Jan-Vitek.atom.xml create mode 100644 blog/feeds/Author-Jan-Vitek.rss.xml create mode 100644 blog/feeds/Author-Jonathan-Schuster.atom.xml create mode 100644 blog/feeds/Author-Jonathan-Schuster.rss.xml create mode 100644 blog/feeds/Author-Julia-Belyakova.atom.xml create mode 100644 blog/feeds/Author-Julia-Belyakova.rss.xml create mode 100644 blog/feeds/Author-Justin-Slepak.atom.xml create mode 100644 blog/feeds/Author-Justin-Slepak.rss.xml create mode 100644 blog/feeds/Author-Kevin-Clancy.atom.xml create mode 100644 blog/feeds/Author-Kevin-Clancy.rss.xml create mode 100644 blog/feeds/Author-Leif-Andersen.atom.xml create mode 100644 blog/feeds/Author-Leif-Andersen.rss.xml create mode 100644 blog/feeds/Author-Li-Yao-Xia.atom.xml create mode 100644 blog/feeds/Author-Li-Yao-Xia.rss.xml create mode 100644 blog/feeds/Author-Matt-Kolosick.atom.xml create mode 100644 blog/feeds/Author-Matt-Kolosick.rss.xml create mode 100644 blog/feeds/Author-Max-New.atom.xml create mode 100644 blog/feeds/Author-Max-New.rss.xml create mode 100644 blog/feeds/Author-Milo-Davis.atom.xml create mode 100644 blog/feeds/Author-Milo-Davis.rss.xml create mode 100644 blog/feeds/Author-Ming-Ho-Yee.atom.xml create mode 100644 blog/feeds/Author-Ming-Ho-Yee.rss.xml create mode 100644 blog/feeds/Author-Olek-Gierczak.atom.xml create mode 100644 blog/feeds/Author-Olek-Gierczak.rss.xml create mode 100644 "blog/feeds/Author-Olivier-Flu\314\210ckiger.atom.xml" create mode 100644 "blog/feeds/Author-Olivier-Flu\314\210ckiger.rss.xml" create mode 100644 blog/feeds/Author-Rob-Kleffner.atom.xml create mode 100644 blog/feeds/Author-Rob-Kleffner.rss.xml create mode 100644 blog/feeds/Author-Sam-Caldwell.atom.xml create mode 100644 blog/feeds/Author-Sam-Caldwell.rss.xml create mode 100644 blog/feeds/Author-Stephen-Chang.atom.xml create mode 100644 blog/feeds/Author-Stephen-Chang.rss.xml create mode 100644 blog/feeds/Author-Tony-Garnock-Jones.atom.xml create mode 100644 blog/feeds/Author-Tony-Garnock-Jones.rss.xml create mode 100644 blog/feeds/Author-William-J-Bowman.atom.xml create mode 100644 blog/feeds/Author-William-J-Bowman.rss.xml create mode 100644 blog/feeds/Author-Zeina-Migeed.atom.xml create mode 100644 blog/feeds/Author-Zeina-Migeed.rss.xml create mode 100644 blog/feeds/CRDTs.atom.xml create mode 100644 blog/feeds/CRDTs.rss.xml create mode 100644 blog/feeds/Emacs.atom.xml create mode 100644 blog/feeds/Emacs.rss.xml create mode 100644 blog/feeds/FFI.atom.xml create mode 100644 blog/feeds/FFI.rss.xml create mode 100644 blog/feeds/Gradual-Typing.atom.xml create mode 100644 blog/feeds/Gradual-Typing.rss.xml create mode 100644 blog/feeds/HOPL.atom.xml create mode 100644 blog/feeds/HOPL.rss.xml create mode 100644 blog/feeds/ICFP.atom.xml create mode 100644 blog/feeds/ICFP.rss.xml create mode 100644 blog/feeds/NEPLS.atom.xml create mode 100644 blog/feeds/NEPLS.rss.xml create mode 100644 blog/feeds/PI-meeting.atom.xml create mode 100644 blog/feeds/PI-meeting.rss.xml create mode 100644 blog/feeds/PL-Junior.atom.xml create mode 100644 blog/feeds/PL-Junior.rss.xml create mode 100644 blog/feeds/PLT-Redex.atom.xml create mode 100644 blog/feeds/PLT-Redex.rss.xml create mode 100644 blog/feeds/Racket.atom.xml create mode 100644 blog/feeds/Racket.rss.xml create mode 100644 "blog/feeds/R\314\214-JIT.atom.xml" create mode 100644 "blog/feeds/R\314\214-JIT.rss.xml" create mode 100644 blog/feeds/Scribble.atom.xml create mode 100644 blog/feeds/Scribble.rss.xml create mode 100644 blog/feeds/System-Administration.atom.xml create mode 100644 blog/feeds/System-Administration.rss.xml create mode 100644 blog/feeds/Takikawa-constant.atom.xml create mode 100644 blog/feeds/Takikawa-constant.rss.xml create mode 100644 blog/feeds/Yoneda.atom.xml create mode 100644 blog/feeds/Yoneda.rss.xml create mode 100644 blog/feeds/about.atom.xml create mode 100644 blog/feeds/about.rss.xml create mode 100644 blog/feeds/adjunction.atom.xml create mode 100644 blog/feeds/adjunction.rss.xml create mode 100644 blog/feeds/all.atom.xml create mode 100644 blog/feeds/all.rss.xml create mode 100644 blog/feeds/announcement.atom.xml create mode 100644 blog/feeds/announcement.rss.xml create mode 100644 blog/feeds/array-language.atom.xml create mode 100644 blog/feeds/array-language.rss.xml create mode 100644 blog/feeds/benchmarking.atom.xml create mode 100644 blog/feeds/benchmarking.rss.xml create mode 100644 blog/feeds/beta.atom.xml create mode 100644 blog/feeds/beta.rss.xml create mode 100644 blog/feeds/books.atom.xml create mode 100644 blog/feeds/books.rss.xml create mode 100644 blog/feeds/bugs.atom.xml create mode 100644 blog/feeds/bugs.rss.xml create mode 100644 blog/feeds/calculus.atom.xml create mode 100644 blog/feeds/calculus.rss.xml create mode 100644 blog/feeds/castle.atom.xml create mode 100644 blog/feeds/castle.rss.xml create mode 100644 blog/feeds/category-theory.atom.xml create mode 100644 blog/feeds/category-theory.rss.xml create mode 100644 blog/feeds/clojure.atom.xml create mode 100644 blog/feeds/clojure.rss.xml create mode 100644 blog/feeds/clojurescript.atom.xml create mode 100644 blog/feeds/clojurescript.rss.xml create mode 100644 blog/feeds/closure-conversion.atom.xml create mode 100644 blog/feeds/closure-conversion.rss.xml create mode 100644 blog/feeds/coYoneda.atom.xml create mode 100644 blog/feeds/coYoneda.rss.xml create mode 100644 blog/feeds/compiler-correctness.atom.xml create mode 100644 blog/feeds/compiler-correctness.rss.xml create mode 100644 blog/feeds/compilers.atom.xml create mode 100644 blog/feeds/compilers.rss.xml create mode 100644 blog/feeds/complete-monitoring.atom.xml create mode 100644 blog/feeds/complete-monitoring.rss.xml create mode 100644 blog/feeds/conference.atom.xml create mode 100644 blog/feeds/conference.rss.xml create mode 100644 blog/feeds/constructions.atom.xml create mode 100644 blog/feeds/constructions.rss.xml create mode 100644 blog/feeds/coq.atom.xml create mode 100644 blog/feeds/coq.rss.xml create mode 100644 blog/feeds/dawn-of-the-digital-era.atom.xml create mode 100644 blog/feeds/dawn-of-the-digital-era.rss.xml create mode 100644 blog/feeds/dear-diary.atom.xml create mode 100644 blog/feeds/dear-diary.rss.xml create mode 100644 blog/feeds/definitions.atom.xml create mode 100644 blog/feeds/definitions.rss.xml create mode 100644 blog/feeds/dissertation.atom.xml create mode 100644 blog/feeds/dissertation.rss.xml create mode 100644 blog/feeds/dsl.atom.xml create mode 100644 blog/feeds/dsl.rss.xml create mode 100644 blog/feeds/dynamic-typing.atom.xml create mode 100644 blog/feeds/dynamic-typing.rss.xml create mode 100644 blog/feeds/event.atom.xml create mode 100644 blog/feeds/event.rss.xml create mode 100644 blog/feeds/eventual-consistency.atom.xml create mode 100644 blog/feeds/eventual-consistency.rss.xml create mode 100644 blog/feeds/extended-abstract.atom.xml create mode 100644 blog/feeds/extended-abstract.rss.xml create mode 100644 blog/feeds/final-encoding.atom.xml create mode 100644 blog/feeds/final-encoding.rss.xml create mode 100644 blog/feeds/galois-connection.atom.xml create mode 100644 blog/feeds/galois-connection.rss.xml create mode 100644 blog/feeds/garbage-collection.atom.xml create mode 100644 blog/feeds/garbage-collection.rss.xml create mode 100644 blog/feeds/ghc.atom.xml create mode 100644 blog/feeds/ghc.rss.xml create mode 100644 blog/feeds/gradual-typing.atom.xml create mode 100644 blog/feeds/gradual-typing.rss.xml create mode 100644 blog/feeds/haskell.atom.xml create mode 100644 blog/feeds/haskell.rss.xml create mode 100644 blog/feeds/higher-order-contracts.atom.xml create mode 100644 blog/feeds/higher-order-contracts.rss.xml create mode 100644 blog/feeds/history.atom.xml create mode 100644 blog/feeds/history.rss.xml create mode 100644 blog/feeds/icfp.atom.xml create mode 100644 blog/feeds/icfp.rss.xml create mode 100644 blog/feeds/instrumentation.atom.xml create mode 100644 blog/feeds/instrumentation.rss.xml create mode 100644 blog/feeds/interactive-syntax.atom.xml create mode 100644 blog/feeds/interactive-syntax.rss.xml create mode 100644 blog/feeds/java.atom.xml create mode 100644 blog/feeds/java.rss.xml create mode 100644 blog/feeds/lambda.atom.xml create mode 100644 blog/feeds/lambda.rss.xml create mode 100644 blog/feeds/lang-extension.atom.xml create mode 100644 blog/feeds/lang-extension.rss.xml create mode 100644 blog/feeds/language-implementation.atom.xml create mode 100644 blog/feeds/language-implementation.rss.xml create mode 100644 blog/feeds/language.atom.xml create mode 100644 blog/feeds/language.rss.xml create mode 100644 blog/feeds/latency.atom.xml create mode 100644 blog/feeds/latency.rss.xml create mode 100644 blog/feeds/lectures.atom.xml create mode 100644 blog/feeds/lectures.rss.xml create mode 100644 blog/feeds/lost-time.atom.xml create mode 100644 blog/feeds/lost-time.rss.xml create mode 100644 blog/feeds/math.atom.xml create mode 100644 blog/feeds/math.rss.xml create mode 100644 blog/feeds/migratory-typing.atom.xml create mode 100644 blog/feeds/migratory-typing.rss.xml create mode 100644 blog/feeds/monotonicity.atom.xml create mode 100644 blog/feeds/monotonicity.rss.xml create mode 100644 blog/feeds/observational-equivalence.atom.xml create mode 100644 blog/feeds/observational-equivalence.rss.xml create mode 100644 blog/feeds/ocaml.atom.xml create mode 100644 blog/feeds/ocaml.rss.xml create mode 100644 blog/feeds/offsite.atom.xml create mode 100644 blog/feeds/offsite.rss.xml create mode 100644 blog/feeds/package.atom.xml create mode 100644 blog/feeds/package.rss.xml create mode 100644 blog/feeds/performance.atom.xml create mode 100644 blog/feeds/performance.rss.xml create mode 100644 blog/feeds/pliss.atom.xml create mode 100644 blog/feeds/pliss.rss.xml create mode 100644 blog/feeds/proceedings.atom.xml create mode 100644 blog/feeds/proceedings.rss.xml create mode 100644 blog/feeds/r.atom.xml create mode 100644 blog/feeds/r.rss.xml create mode 100644 blog/feeds/racket.atom.xml create mode 100644 blog/feeds/racket.rss.xml create mode 100644 blog/feeds/reduction.atom.xml create mode 100644 blog/feeds/reduction.rss.xml create mode 100644 blog/feeds/scope.atom.xml create mode 100644 blog/feeds/scope.rss.xml create mode 100644 blog/feeds/semantics.atom.xml create mode 100644 blog/feeds/semantics.rss.xml create mode 100644 blog/feeds/statistics.atom.xml create mode 100644 blog/feeds/statistics.rss.xml create mode 100644 blog/feeds/transient.atom.xml create mode 100644 blog/feeds/transient.rss.xml create mode 100644 blog/feeds/turnstile.atom.xml create mode 100644 blog/feeds/turnstile.rss.xml create mode 100644 blog/feeds/tutorial.atom.xml create mode 100644 blog/feeds/tutorial.rss.xml create mode 100644 blog/feeds/tutorials.atom.xml create mode 100644 blog/feeds/tutorials.rss.xml create mode 100644 blog/feeds/typed-racket.atom.xml create mode 100644 blog/feeds/typed-racket.rss.xml create mode 100644 blog/feeds/types.atom.xml create mode 100644 blog/feeds/types.rss.xml create mode 100644 blog/feeds/visr.atom.xml create mode 100644 blog/feeds/visr.rss.xml create mode 100644 blog/feeds/windows.atom.xml create mode 100644 blog/feeds/windows.rss.xml create mode 100644 blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png create mode 100644 blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png create mode 100644 blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png create mode 100644 blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png create mode 100644 blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png create mode 100644 blog/index-2.html create mode 100644 blog/index-3.html create mode 100644 blog/index-4.html create mode 100644 blog/index-5.html create mode 100644 blog/index-6.html create mode 100644 blog/index-7.html create mode 100644 blog/index-8.html create mode 100644 blog/index-9.html create mode 100644 blog/index.html create mode 100644 blog/sitemap.txt create mode 100644 blog/tags/1st-blog-post.html create mode 100644 blog/tags/Author-Aaron-Weiss.html create mode 100644 blog/tags/Author-Alexi-Turcotte.html create mode 100644 blog/tags/Author-Artem-Pelenitsyn.html create mode 100644 blog/tags/Author-Asumu-Takikawa.html create mode 100644 blog/tags/Author-Ben-Chung.html create mode 100644 blog/tags/Author-Ben-Greenman-2.html create mode 100644 blog/tags/Author-Ben-Greenman-3.html create mode 100644 blog/tags/Author-Ben-Greenman-4.html create mode 100644 blog/tags/Author-Ben-Greenman.html create mode 100644 blog/tags/Author-Daniel-Patterson.html create mode 100644 blog/tags/Author-Dustin-Jamner.html create mode 100644 blog/tags/Author-Gabriel-Scherer-2.html create mode 100644 blog/tags/Author-Gabriel-Scherer.html create mode 100644 blog/tags/Author-Jan-Vitek.html create mode 100644 blog/tags/Author-Jonathan-Schuster.html create mode 100644 blog/tags/Author-Julia-Belyakova.html create mode 100644 blog/tags/Author-Justin-Slepak.html create mode 100644 blog/tags/Author-Kevin-Clancy.html create mode 100644 blog/tags/Author-Leif-Andersen.html create mode 100644 blog/tags/Author-Li-Yao-Xia.html create mode 100644 blog/tags/Author-Matt-Kolosick.html create mode 100644 blog/tags/Author-Max-New.html create mode 100644 blog/tags/Author-Milo-Davis.html create mode 100644 blog/tags/Author-Ming-Ho-Yee.html create mode 100644 blog/tags/Author-Olek-Gierczak.html create mode 100644 "blog/tags/Author-Olivier-Flu\314\210ckiger.html" create mode 100644 blog/tags/Author-Rob-Kleffner.html create mode 100644 blog/tags/Author-Sam-Caldwell.html create mode 100644 blog/tags/Author-Stephen-Chang.html create mode 100644 blog/tags/Author-Tony-Garnock-Jones.html create mode 100644 blog/tags/Author-William-J-Bowman.html create mode 100644 blog/tags/Author-Zeina-Migeed.html create mode 100644 blog/tags/CRDTs.html create mode 100644 blog/tags/Emacs.html create mode 100644 blog/tags/FFI.html create mode 100644 blog/tags/Gradual-Typing.html create mode 100644 blog/tags/HOPL-2.html create mode 100644 blog/tags/HOPL.html create mode 100644 blog/tags/ICFP.html create mode 100644 blog/tags/NEPLS.html create mode 100644 blog/tags/PI-meeting.html create mode 100644 blog/tags/PL-Junior.html create mode 100644 blog/tags/PLT-Redex.html create mode 100644 blog/tags/Racket.html create mode 100644 "blog/tags/R\314\214-JIT.html" create mode 100644 blog/tags/Scribble.html create mode 100644 blog/tags/System-Administration.html create mode 100644 blog/tags/Takikawa-constant.html create mode 100644 blog/tags/Yoneda.html create mode 100644 blog/tags/about.html create mode 100644 blog/tags/adjunction.html create mode 100644 blog/tags/announcement.html create mode 100644 blog/tags/array-language.html create mode 100644 blog/tags/benchmarking.html create mode 100644 blog/tags/beta.html create mode 100644 blog/tags/books.html create mode 100644 blog/tags/bugs.html create mode 100644 blog/tags/by-Aaron-Weiss.html create mode 100644 blog/tags/by-Alexi-Turcotte.html create mode 100644 blog/tags/by-Artem-Pelenitsyn.html create mode 100644 blog/tags/by-Asumu-Takikawa.html create mode 100644 blog/tags/by-Ben-Chung.html create mode 100644 blog/tags/by-Ben-Greenman-2.html create mode 100644 blog/tags/by-Ben-Greenman-3.html create mode 100644 blog/tags/by-Ben-Greenman-4.html create mode 100644 blog/tags/by-Ben-Greenman.html create mode 100644 blog/tags/by-Daniel-Patterson.html create mode 100644 blog/tags/by-Dustin-Jamner.html create mode 100644 blog/tags/by-Gabriel-Scherer-2.html create mode 100644 blog/tags/by-Gabriel-Scherer.html create mode 100644 blog/tags/by-Jan-Vitek.html create mode 100644 blog/tags/by-Jonathan-Schuster.html create mode 100644 blog/tags/by-Julia-Belyakova.html create mode 100644 blog/tags/by-Justin-Slepak.html create mode 100644 blog/tags/by-Kevin-Clancy.html create mode 100644 blog/tags/by-Leif-Andersen.html create mode 100644 blog/tags/by-Li-Yao-Xia.html create mode 100644 blog/tags/by-Matt-Kolosick.html create mode 100644 blog/tags/by-Max-New.html create mode 100644 blog/tags/by-Milo-Davis.html create mode 100644 blog/tags/by-Ming-Ho-Yee.html create mode 100644 blog/tags/by-Olek-Gierczak.html create mode 100644 blog/tags/by-Rob-Kleffner.html create mode 100644 blog/tags/by-Sam-Caldwell.html create mode 100644 blog/tags/by-Stephen-Chang.html create mode 100644 blog/tags/by-Tony-Garnock-Jones.html create mode 100644 blog/tags/by-William-J-Bowman.html create mode 100644 blog/tags/by-Zeina-Migeed.html create mode 100644 blog/tags/calculus.html create mode 100644 blog/tags/castle.html create mode 100644 blog/tags/category-theory.html create mode 100644 blog/tags/clojure.html create mode 100644 blog/tags/clojurescript.html create mode 100644 blog/tags/closure-conversion.html create mode 100644 blog/tags/coYoneda.html create mode 100644 blog/tags/compiler-correctness.html create mode 100644 blog/tags/compilers.html create mode 100644 blog/tags/complete-monitoring.html create mode 100644 blog/tags/conference.html create mode 100644 blog/tags/constructions.html create mode 100644 blog/tags/coq.html create mode 100644 blog/tags/dawn-of-the-digital-era.html create mode 100644 blog/tags/dear-diary.html create mode 100644 blog/tags/definitions.html create mode 100644 blog/tags/dissertation.html create mode 100644 blog/tags/dsl.html create mode 100644 blog/tags/dynamic-typing.html create mode 100644 blog/tags/event.html create mode 100644 blog/tags/eventual-consistency.html create mode 100644 blog/tags/extended-abstract.html create mode 100644 blog/tags/final-encoding.html create mode 100644 blog/tags/galois-connection.html create mode 100644 blog/tags/garbage-collection.html create mode 100644 blog/tags/ghc.html create mode 100644 blog/tags/gradual-typing.html create mode 100644 blog/tags/haskell.html create mode 100644 blog/tags/higher-order-contracts.html create mode 100644 blog/tags/history.html create mode 100644 blog/tags/icfp.html create mode 100644 blog/tags/instrumentation.html create mode 100644 blog/tags/interactive-syntax.html create mode 100644 blog/tags/java.html create mode 100644 blog/tags/lambda.html create mode 100644 blog/tags/lang-extension.html create mode 100644 blog/tags/language-implementation.html create mode 100644 blog/tags/language.html create mode 100644 blog/tags/latency.html create mode 100644 blog/tags/lectures.html create mode 100644 blog/tags/lost-time.html create mode 100644 blog/tags/math.html create mode 100644 blog/tags/migratory-typing.html create mode 100644 blog/tags/monotonicity.html create mode 100644 blog/tags/observational-equivalence.html create mode 100644 blog/tags/ocaml.html create mode 100644 blog/tags/offsite.html create mode 100644 blog/tags/package.html create mode 100644 blog/tags/performance.html create mode 100644 blog/tags/pliss.html create mode 100644 blog/tags/proceedings.html create mode 100644 blog/tags/r.html create mode 100644 blog/tags/racket.html create mode 100644 blog/tags/reduction.html create mode 100644 blog/tags/scope.html create mode 100644 blog/tags/semantics.html create mode 100644 blog/tags/statistics.html create mode 100644 blog/tags/transient.html create mode 100644 blog/tags/turnstile.html create mode 100644 blog/tags/tutorial.html create mode 100644 blog/tags/tutorials.html create mode 100644 blog/tags/typed-racket.html create mode 100644 blog/tags/types.html create mode 100644 blog/tags/visr.html create mode 100644 blog/tags/windows.html create mode 100644 contact.html create mode 100644 index.html create mode 100644 new-members.html create mode 100644 people.html create mode 100644 publications.html create mode 100644 seminars.html create mode 100644 software.html create mode 100644 teaching.html diff --git a/blog/.frog/build b/blog/.frog/build new file mode 100644 index 00000000..0c20de54 --- /dev/null +++ b/blog/.frog/build @@ -0,0 +1 @@ +((3) 0 () 88 ((p+ #"/home/runner/work/website/website/blog/_src/posts/2022-01-06-introducing-visr-md.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-06-07-icfp-2016-looking-for-student-volunteers.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-03-03-metafunction-apply.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-05-04-rank-polymorphism.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-01-17-how-to-prove-a-compiler-correct.md" . unix) (u . "") (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-01-19-untyped-programs-don-t-exist.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-10-11-compcert-overview.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-06-16-spring-2017-pl-junior-retrospective.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-07-17-continuations-1983.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-06-24-turing-award-50.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-09-05-lexical-and-dynamic-scope.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-05-01-hopl-dyn-cats.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-11-02-beta-reduction-part-1.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-02-28-pliss-oregon-without-the-greek.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-04-04-top-5-in-last-50.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-09-27-final-algebra-semantics-is-observational-equivalence.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-11-16-monotonicity-types-towards-a-type-system-for-eventual-consistency.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-09-25-redex-faq.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-04-12-making-an-ide-plugin-for-drracket.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2022-02-16-rsh-overview-1.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-03-24-what-even-is-compiler-correctness.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-05-15-artifacts-for-semantics.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-05-09-hopl-no-good-answers.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-06-29-tutorial-racket-ffi-part-2.scrbl" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-02-21-bullets-are-good-for-your-coq-proofs.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-02-28-hopl2017-linear-types.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-06-13-does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-06-27-tutorial-using-racket-s-ffi.scrbl" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-02-15-hopl2017-conversational-context.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-02-21-datalog-for-static-analysis.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-04-07-forgetful-and-heedful-contracts.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-05-11-conversational-concurrency.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-11-30-turnstile-community.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-10-22-defining-local-bindings-in-turnstile-languages.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-11-24-disappearing-code.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-11-30-getting-started-in-programming-languages-cross-post.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-04-23-how-to-prove-a-compiler-fully-abstract.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-09-10-scoping-in-r.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-03-09-pliss-learn-about-pl-implementation-in-a-castle.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-05-03-nepls-on-may-31st-at-umass-amherst.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-11-16-constructive-galois-connections.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-11-17-src-submissions.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-12-17-measuring-the-submission-review-balance.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2020-11-12-transient-opt-kw-lambda.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2020-12-23-deep-and-shallow-types.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-08-03-a-few-cores-too-many.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-09-15-nepls-on-october-7th-at-northeastern.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-06-05-trip-report-pliss-2017.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-06-09-bridging-the-system-configuration-gap.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-05-24-plc-russia.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-05-23-scribble-html.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-07-25-tutorial-zero-to-sixty-in-racket.scrbl" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-07-11-tutorial-racket-ffi-part-3.scrbl" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-02-17-scribble-acmart.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-10-19-history-of-actors.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-10-17-emacs-daemon-for-fast-editor-startup.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-10-31-meaningful-distinctions.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-08-22-gradual-typing-across-the-spectrum.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-04-27-racket-school-2018.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-01-02-fall-2016-pl-junior-retrospective.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-02-16-hopl2017-intro.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-12-02-java-transient.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-12-12-prl-offsite-2019.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-08-13-reviews-and-author-responses.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-07-19-trees-1973.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-04-29-welcome-to-the-prl-blog.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2016-05-18-gradual-typing-across-the-spectrum.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-10-31-complete-monitors-for-gradual-types.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-09-10-four-kinds-of-scoping-in-r.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2020-10-15-shallow-mailing-list.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2020-01-15-tr-optimizer-vs-transient.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-08-29-why-am-i-going-to-icfp-2017.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-08-28-closure-conversion-coyoneda.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2019-01-28-on-stack-replacement.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-12-11-the-behavior-of-gradual-types.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-10-06-a-spectrum-of-type-soundness-and-performance.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2018-05-08-sampling-gradual-typing-performance.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-05-26-racket-6-9-and-windows-10-creators-update.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-06-05-syntactic-parametricity-strikes-again.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-03-15-tracing-jits-for-dynamic-languages.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-03-10-hopl-stack.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-04-20-hopl-refinement-types.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-04-17-hopl-types-in-compilation.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-04-25-prl-at-snapl-17.md" . unix) (p+ #"/home/runner/work/website/website/blog/_src/posts/2017-04-28-hopl-soft-typing.md" . unix)) () (h ! (equal) ((? . 0) f post (u . "Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript)") (? . 0) 1731622765 (p+ #"/home/runner/work/website/website/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/index.html" . unix) (u . "/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/") (u . "2022-01-06T17:56:08") (? . 45) (? . 20) (c (u . "visr") c (u . "clojure") c (u . "clojurescript") c (u . "interactive syntax") c (u . "Author: Leif Andersen")) (u . "\n

\n

\n\n

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

\n\n
\"Graphical\n

Graphical network embedded in text

\n\n

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

\n\n\n\n

To learn more about interactive-syntax, watch this video or read the accompanying paper.

\n\n\n\n

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

\n\n

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

") #t (u . "\n

\n

\n\n

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

\n\n
\"Graphical\n

Graphical network embedded in text

\n\n

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

\n\n\n\n

To learn more about interactive-syntax, watch this video or read the accompanying paper.

\n\n\n\n

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

\n\n

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

\n\n\n

Getting started with VISr

\n\n

Start by going to visr.pl, which is a web-based IDE that directly supports VISrs. Once in the IDE, press Insert VISr to place a VISr at the current cursor position. This VISr contains two buttons:

\n\n\n\n
\"VISr\"\n

VISr

\n\n

Opening the code shows that the new VISr is an instance of visr.core/empty-visr, a default VISr provided by the IDE. This VISr expects a map with the key :message to display in the visual view. Changing the value associated with :message changes what is displayed, in this case “Endless Possibility”:

\n\n
\"Open\n

Open Visr

\n\n

Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:

\n\n
\n \n \n \n \n
\n
\n 
1\n2
\n
\n 
^{:editor visr.core/empty-visr}(visr.core/empty-visr+elaborate \n                                 {:message \"Endless Possibility\"})\n
\n
\n
\n\n

This operation works in reverse too. Writing out similar text and pasting it into visr.pl yields its visual representation.

\n\n

Making a new VISr

\n\n

The defvisr form creates a VISr type. This form expects two methods:

\n\n
    \n
  1. a render method that provides visualization and interaction when code is edited, and
    2. \n
  2. an elaborate/elaborate-fn method that gives the VISr compile-time and run-time semantics.
\n\n

The following is the signature for a simple VISr type:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
(ns example.core)\n\n(defvisr Counter\n  (elaborate-fn [this] \"TODO-elaborate\")\n  (render [this] \"TODO-render\"))\n
\n
\n
\n\n

This example uses elaborate-fn, a simplified version of elaborate that gives the VISr the same semantics as a function application. It also allows the defvisr form to work in the same file as the VISr itself.

\n\n
\"Example\n

Example of elaborate-fn semantics

\n\n

The Render Method for Edit-Time Semantics

\n\n

The render method is given the VISr state as an atom; updating this atom also updates the code to reflect the new state. The return value for render must be a Reagent form that is the visual view for the VISr. A render method for a counter VISr might look as follows:

\n\n
\n \n \n \n \n
\n
\n 
1\n2
\n
\n 
(render [this]\n  [:button {:on-click #(swap! this inc)} @this])\n
\n
\n
\n\n

And in action:

\n\n
\"Simple\n

Simple Count Example

\n\n

This VISr doesn’t match the theme of the page; it also requires the state to be a single number. Using React Bootstrap and Reagent cursors fixes both of these issues:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10
\n
\n 
(ns example.core\n  (:require [reagent.core :refer [cursor]]\n            [react-bootstrap :refer [Button]]))\n            \n(defvisr Counter\n  (elaborate-fn [this] \"TODO-elaborate\")\n  (render [this]\n    (let [count (cursor this [:count])]\n      (when-not @count (reset! count 0))\n      [:> Button {:on-click #(swap! count inc)} @count])))\n
\n
\n
\n\n

Elaboration and Run-Time Semantics

\n\n

The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of elaborate-fn, the VISr semantics takes the form of a function application:

\n\n
\n \n \n \n \n
\n
\n 
1
\n
\n 
(elaborate-fn [{:keys [count]}] count)\n
\n
\n
\n\n

This elaborate method expects a dictionary with the key :count and returns the value associated with that key. It makes use of ClojureScript’s Destructuring for brevity. The following code is equivalent:

\n\n
\n \n \n \n \n
\n
\n 
1
\n
\n 
(elaborate-fn [this] (get this :count))\n
\n
\n
\n\n

Putting it all together

\n\n

The final result is:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10
\n
\n 
(ns test.core\n  (:require [reagent.core :refer [cursor]]\n            [react-bootstrap :refer [Button]]))\n\n(defvisr Counter\n  (elaborate-fn [{:keys [count]}] count)\n  (render [this]\n    (let [count (cursor this [:count])]\n      (when-not @count (reset! count 0))\n      [:> Button {:on-click #(swap! count inc)} @count])))\n
\n
\n
\n\n

Here is the VISr in action:

\n\n
\"Full\n

Full Count Example

\n\n

That’s all there is to it. From here, you can go to visr.pl to make your own programs using VISr. You can also take this survey, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to the visr project page.

\n\n

Thanks for reading, happy coding!

")) ((? . 1) f post (u . "ICFP 2016: looking for student volunteers") (? . 1) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/06/07/icfp-2016-looking-for-student-volunteers/index.html" . unix) (u . "/blog/2016/06/07/icfp-2016-looking-for-student-volunteers/") (u . "2016-06-07T11:53:47") (? . 68) (? . 27) (c (u . "ICFP") c (u . "Author: Gabriel Scherer")) (u . "\n

If you are a student, you should consider applying to become an ICFP 2016 student volunteer! The deadline for application is July 31st, 2016.

") #t (u . "\n

If you are a student, you should consider applying to become an ICFP 2016 student volunteer! The deadline for application is July 31st, 2016.

\n\n\n

ICFP 2016, the Internal Conference on Functional Programming, is happening in Nara, Japan. If you are a student, you may be interest in being a Student Volunteer: you help run the conference, and in exchange do not pay registration fees — but you still have to find funding for the travel, hosting, and dinners. Quoting the Student Volunteer webpage:

\n\n
\n

ICFP is pleased to offer a number of opportunities for student volunteers, who are vital to the efficient operation and continued success of the conference each year. The student volunteer program is a chance for students from around the world to participate in the conferences whilst assisting us in preparing and running the event.

\n

Job assignments for student volunteers include assisting with technical sessions, workshops, tutorials and panels, helping the registration desk, operating the information desk, helping with traffic flow, and general assistance to keep the conferences running smoothly.

\n

The Student Volunteer Program helps more students attend the ICFP conference by covering conference fees (but not travel or lodging expenses) in exchange for a fixed number of work hours (usually from 8 to 12) helping with the conference organization (registration and information desks, assistance during talk sessions, etc.).

\n

The Student Volunteer registration covers:

\n \n

To apply, please fill the following form.

\n

The application deadline is July 31st, 2016. Applications after this date may be considered pending availability.

\n

You can send questions about student volunteering to icfp-SV at researchr dot org.

\n\n

I was “student volunteer captain” at ICFP last year in Vancouver, and I will do it again this year. My entirely personal take on the thing is that being a Student Volunteer is worth it, but that being a Captain is too much work.

\n\n

The main downside of being a volunteer is some of the shifts are at the registration desk, and they may imply missing some of the talks — and also you may have to get up early for your duties. The upsides are many. You get belong to a small group of nice people. You have interactions with many people without much effort; you will enjoy the sparks of gratitude in the eyes of the “Where is Workshop Room B2?” crowd. You have a small peek into the kind of work involved in running a conference; most people actually working on the organization (we SVs are hobbyists) are pouring surprising amount of work. Also, you learn to fold tee-shirts very well, if you’re on “bag stuffing” duty.

\n\n

Being a student volunteer can be combined with other forms of travel support, such as SIGPLAN PAC funding; see the travel support page for more details.

\n\n

Another thing you should think about is applying to PLMW, the Programming Languages Mentoring Workshop that happens at ICFP, POPL, and PLDI. PLMW funding covers the whole conference cost (travel, housing, registration, dinners), so if you get PLMW funding you have no financial motivation to be a student volunteer. This year, PLMW focuses on early career graduate students.

")) ((? . 2) f post (u . "PLT Redex: mf-apply") (? . 2) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/03/03/plt-redex-mf-apply/index.html" . unix) (u . "/blog/2017/03/03/plt-redex-mf-apply/") (u . "2017-03-03T08:54:20") (? . 14) (? . 83) (c (u . "PLT Redex") c (u . "package") c (u . "lang-extension") c (u . "Author: Ben Greenman")) (u . "\n

The mf-apply keyword is for checked metafunction application in PLT Redex. In other words, (mf-apply f x) is just like (f x), but errors if f is not a previously-defined metafunction.

\n\n

Also, consider applying to attend The Racket School of Semantics and Languages in Salt Lake City this summer: http://summer-school.racket-lang.org/2017/

") #t (u . "\n

The mf-apply keyword is for checked metafunction application in PLT Redex. In other words, (mf-apply f x) is just like (f x), but errors if f is not a previously-defined metafunction.

\n\n

Also, consider applying to attend The Racket School of Semantics and Languages in Salt Lake City this summer: http://summer-school.racket-lang.org/2017/

\n\n\n

Metafunctions vs. List Patterns

\n\n

Have you used PLT Redex? Good! Maybe this has happened to you:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n53\n54\n55\n56\n57\n58\n59\n60\n61\n62\n63\n64\n65\n66\n67\n68\n69\n70
\n
\n 
#lang racket\n(require redex)\n\n;; -----------------------------------------------------------------------------\n;; 1. You define a language\n(define-language STLC\n  [V ::= integer boolean C]\n  [C ::= (closure Λ ρ)]\n  [Λ ::= (λ (x : τ) M)]\n  [M ::= (M M) V Λ x]\n  [τ ::= Int Bool (τ → τ)]\n  [ρ ::= ((x V) ...)]\n  [Γ ::= ((x τ) ...)]\n  [x ::= variable-not-otherwise-mentioned]\n  #:binding-forms (λ (x : τ) M #:refers-to x))\n\n\n;; -----------------------------------------------------------------------------\n;; 2. You define a few metafunctions\n(define-metafunction STLC\n  closure->lam : C -> Λ\n  [(closure->lam (closure Λ ρ))\n   Λ])\n\n(define-metafunction STLC\n  closure->env : C -> ρ\n  [(closure->env (closure Λ ρ))\n   ρ])\n\n\n;; -----------------------------------------------------------------------------\n;; 3. You try defining a judgment form . . .\n(define-judgment-form STLC\n  #:mode (free-variables I O)\n  #:contract (free-variables M (x ...))\n  [\n   --- FVS-Var\n   (free-variables x (x))]\n  [\n   (free-variables M_0 (x_0 ...))\n   (free-variables M_1 (x_1 ...))\n   --- FVS-App\n   (free-variables (M_0 M_1) (x_0 ... x_1 ...))]\n  [\n   (where (λ (x_0 τ) M) Λ)\n   (free-variables M (x_1 ...))\n   (where (x_2 ...) ,(set-remove (term (x_1 ...)) (term x_0)))\n   --- FVS-Λ\n   (free-variables Λ (x_2 ...))]\n  [\n   --- FVS-Integer\n   (free-variables integer_0 ())]\n  [\n   --- FVS-Boolean\n   (free-variables boolean_0 ())]\n  [\n   (where Λ (closure->lam C))\n   (free-variables Λ (x_0 ...))\n   (where ((x_1 τ_1) ...) (closure-env C))\n   (where (x_2 ...) ,(set-subtract (term (x_0 ...)) (term (x_1 ...))))\n   --- FVS-Closure\n   (free-variables C (x_2 ...))])\n\n\n;; -----------------------------------------------------------------------------\n;; 4. You test the judgment, and it mysteriously fails\n(judgment-holds\n  (free-variables (closure (λ (x : Int) x) ())\n                  ()))\n;; ==> #f\n
\n
\n
\n\n

WHAT HAPPENED??!

\n\n

The problem is this line in the FVS-Closure rule:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
   ....\n   (where ((x_1 τ_1) ...) (closure-env C))\n   ....\n
\n
\n
\n\n

which checks that the list (closure-env C) (whose first element is the symbol closure-env and second element is the symbol C) matches the pattern ((x_1 τ_1) ...).

\n\n

Right.

\n\n

Of course you meant to apply the metafunction closure->env but made a typo. And since the syntax for metafunction application is the same as the syntax for building a list, Redex doesn’t report an error.

\n\n

We can fix this code with the new mf-apply keyword (available on GitHub or in a snapshot build):

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
   ....\n   (where ((x_1 τ_1) ...) (mf-apply closure-env C))\n   ....\n
\n
\n
\n\n

Running raco make now gives a compile-time error.

\n\n
  term: expected a previously defined metafunction\n    at: closure-env\n    in: (mf-apply closure-env C)
\n\n

But I still need to type mf-apply correctly!

\n\n

Leif Andersen says:

\n\n
\n

I should point out that this has the issue of you still need to type mf-apply correctly. ;)

\n\n

That is, if you accidentally write:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
   ....\n   (where ((x_1 τ_1) ...) (mf-applu closure-env C))\n   ....\n
\n
\n
\n\n

Then the code compiles, thinking you intend to match a list of three elements against the pattern.

\n\n

Never fear, there are at least two solutions.

\n\n

Solution 1: rename mf-apply

\n\n

A simple fix is to rename the mf-apply keyword to something shorter (and harder to mis-type):

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4
\n
\n 
#lang racket\n(require redex\n         (rename-in redex\n           [mf-apply MF]))\n
\n
\n
\n\n

Solution 2: the mf-apply lang extension

\n\n

A fancier solution is to install the mf-apply meta-language.

\n\n
  $ raco pkg install mf-apply
\n\n

This language updates the readtable to interpret S-expressions that begin with #{:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4
\n
\n 
#lang mf-apply racket\n(require redex)\n\n(term #{f x ...})\n
\n
\n
\n\n

as a metafunction application:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4
\n
\n 
#lang mf-apply racket\n(require redex)\n\n(term (mf-apply f x ...))\n
\n
\n
\n\n

You the programmer only needs to write the #{....} syntax.

\n\n

Source code is on GitHub:

\n\n\n\n

(It’s the simplest lang-extension I know of)

\n\n

What is PLT Redex?

\n\n

PLT Redex is a library for semantics engineering. One of my favorite Redex features is it implements capture-avoiding substitution and α-equivalence for any language with a #:binding-forms specification (such as STLC, above).

\n\n

You can read more:

\n\n\n\n

And if you act now, you can become a Redexan between July 10 and July 14 at the summer school in Salt Lake City, Utah:

\n\n")) ((? . 3) f post (u . "Rank Polymorphism") (? . 3) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/05/04/rank-polymorphism/index.html" . unix) (u . "/blog/2017/05/04/rank-polymorphism/") (u . "2017-05-04T18:26:48") (? . 12) (? . 23) (c (u . "array language") c (u . "Author: Justin Slepak")) (u . "\n

Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let’s just call it lerp) for scalars:

\n\n
(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))
\n\n

The number marks on each argument indicate the expected “rank” of the argument: how many dimensions it should have. In this case, each one is marked 0, indicating a scalar (i.e., 0-dimensional) argument. The function is usable as-is for

\n\n") #t (u . "\n

Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let’s just call it lerp) for scalars:

\n\n
(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))
\n\n

The number marks on each argument indicate the expected “rank” of the argument: how many dimensions it should have. In this case, each one is marked 0, indicating a scalar (i.e., 0-dimensional) argument. The function is usable as-is for

\n\n\n\n\n

Each of these use cases mixes the argument dimensions a little differently. A pixel is a vector (a rank–1 structure) of numbers representing color channel values, so the α-blending case uses two vector arguments and one scalar argument.

\n\n

The only real difference between these use cases is the iteration space: they’re all effectively loop nests around the same basic scalar operation. In a rank-polymorphic language, the iteration space is derived automatically from the data, so you don’t need to write out the control structure yourself.

\n\n

The fundamental idea behind function application here is breaking the argument arrays into lower-ranked pieces called “cells.” Each cell has the rank expected by the function being applied. In the case of lerp, the pixels, images, videos, etc. are all broken up into rank–0 (scalar) cells because lerp expects rank–0 arguments. Other expected ranks are possible as well— a vector dot product function dot-prod would call for rank–1 cells, and a matrix inversion function minv would call for rank–2 cells.

\n\n

The structure built up around the cells is called the “frame.” A matrix array is a rank–2 frame containing rank–0 cells for lerp, but it would be a rank–1 frame containing rank–1 cells for dot-prod and a rank–0 frame containing a single rank–1 cell for minv. A rank-n array could be broken down into a frame of cells in n+1 different ways, and it’s the function being applied that determines which decomposition to use.

\n\n

Unfortunately, the implicit control structure that’s so convenient for the programmer is a problem for a compiler. Historically, implementations of such languages have had to do without static information about the iteration space. Interpreters (and line-at-a-time compilers, to a lesser extent) get to inspect the concrete data they’re dealing with, but static compilers have had to make do with emitting a generic loop structure. A “loop” over a scalar might sound like trivial overhead, but not when it appears within some other hot loop. Being unable to see when loop boundaries match up is also a barrier to loop fusion. The lack of thorough static shape information was a long-standing problem my advisor pointed out to me when I was a new student looking at possible research projects, and he was interested in applying some form of static analysis to gather that information.

\n\n

The first step in addressing it was to come up with a formal semantics for rank polymorphism. Although APL has existed since the 1960s, it had mostly lived in a separate world from mainstream programming language research. The formal techniques developed in PL had seen little to no application to APL and its “successor” language J.

\n\n

There’s a lot to dislike about APL and J—special case behavior in many of the primitive operators, limited function arity, syntactic separation of first-order and second-order functions, the impossibility of parsing an entire program at once (fun fact: static analysis has been tried there)—and of course the idiosyncratic identifiers used for primops have prompted plenty of internet arguments. None of those things are essential to the programming model, so I’m building a new language called Remora to isolate the aspects I want to study.

\n\n

People don’t always think of a type system as a form of static analysis, but it turned out to be an effective way of gathering shape information. Remora’s type system uses a restricted form of dependent types, in the style of Dependent ML. An array type is indexed by the shape, the numeric sizes of the array’s individual dimensions. Index polymorphism (i.e., Π types) allows functions to work on varying cell shapes and even varying cell ranks (which is essential for primitives like append and reduce, which operate along the major axis of arrays, no matter their rank). Frame-rank polymorphism, which gives rise to the control structure, remains completely implicit, leaving it to be identified by the type rule for function application. As a nice bonus, type soundness rules out run-time errors arising from incompatible argument shapes.

\n\n
\n\n

If you liked this post, you may also be interested in:

\n\n")) ((? . 4) f post (u . "[How to prove a compiler correct (cross-post)](https://dbp.io/essays/2018-01-16-how-to-prove-a-compiler-correct.html)") (? . 4) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/01/17/-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html/index.html" . unix) (u . "/blog/2018/01/17/-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html/") (u . "2018-01-17T20:58:48") (? . 17) (? . 6) (c (u . "Author: Daniel Patterson")) (? . 5) #f (? . 5)) ((? . 6) f post (u . "[Untyped Programs Don't Exist (cross-post)](https://williamjbowman.com/blog/2018/01/19/untyped-programs-don-t-exist/)") (? . 6) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/01/19/-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist/index.html" . unix) (u . "/blog/2018/01/19/-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist/") (u . "2018-01-19T17:05:00") (? . 4) (? . 19) (c (u . "Author: William J. Bowman")) (? . 5) #f (? . 5)) ((? . 7) f post (u . "CompCert Overview") (? . 7) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/10/11/compcert-overview/index.html" . unix) (u . "/blog/2016/10/11/compcert-overview/") (u . "2016-10-11T17:41:16") (? . 47) (? . 56) (c (u . "tutorial") c (u . "coq") c (u . "compiler correctness") c (u . "Author: Ben Greenman")) (u . "\n

If you are interested in learning about the internals of the CompCert C compiler but would rather not read its source code, this post is for you.

") #t (u . "\n

If you are interested in learning about the internals of the CompCert C compiler but would rather not read its source code, this post is for you.

\n\n\n

(This is a public service announcement.)

\n\n

Last fall, I gave a short lecture on the 2006 paper “Formal Certification of a Compiler Back-End” by Xavier Leroy for Amal Ahmed’s “Special Topics in Programming Languages” class. Rather than present CompCert as it existed in 2006, I read the documentation and source code for CompCert 2.5 (released June 2015). The lecture then focused on three questions:

\n\n\n\n

My notes for the lecture give a “mid-level” summary of the compiler — there are more details than you’ll find in papers, but it’s (hopefully!) easier to read than the source code. The document is also hyperlinked to locations in the CompCert GitHub repository.

\n\n

Here is the document:

\n\n
\n

http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf

\n\n

And here is a table-of-contents:

\n\n
    \n
  1. Motivation, details of the source and target languages, high-level guarantees
    2. \n
  2. Compiler pipeline, optimizing passes, links intermediate language grammars and Coq theorems
    3. \n
  3. Background on compiler correctness
    4. \n
  4. CompCert’s correctness, properties that CompCert does not guarantee
    5. \n
  5. Recent (2006 – 2015) work in the CompCert ecosystem
\n\n

The document ends with a short description of two other research projects that have grown into “industry software” and a link to Xaver Leroy’s OPLSS lectures on certified compilers. Enjoy!

")) ((? . 8) f post (u . "Spring 2017 PL Junior Retrospective") (? . 8) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/06/16/spring-2017-pl-junior-retrospective/index.html" . unix) (u . "/blog/2017/06/16/spring-2017-pl-junior-retrospective/") (u . "2017-06-16T11:38:25") (? . 49) (? . 10) (c (u . "PL Junior") c (u . "Author: Ben Chung") c (u . "Author: Milo Davis") c (u . "Author: Ming-Ho Yee") c (u . "Author: Matt Kolosick") c (u . "Author: Dustin Jamner") c (u . "Author: Artem Pelenitsyn") c (u . "Author: Julia Belyakova") c (u . "Author: Sam Caldwell")) (u . "\n

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

\n\n

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

") #t (u . "\n

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

\n\n

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

\n\n\n

Format

\n\n

As discussed in last fall’s retrospective, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.

\n\n

One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.

\n\n

In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!

\n\n

Synthesis

\n\n

The first paper on program synthesis that we looked at was A Deductive Approach to Program Synthesis by Manna and Waldinger. We chose this paper because it’s old and has a lot of citations so it’s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.

\n\n

The next paper was Programmatic and Direct Manipulation, Together by Chugh, Hempel, Spradlin, and Alders, which presents the Sketch-n-Sketch system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.

\n\n

We concluded our program synthesis segment with Type-and-Example-Directed Program Synthesis by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.

\n\n

While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior

\n\n

Sequent Calculus

\n\n

We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by Herbelin. Gabriel’s thesis (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.

\n\n

It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.

\n\n

An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.

\n\n

Logic Programming

\n\n

The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what cut does).

\n\n

The second session presented the paper The Semantics of Predicate Logic as a Programming Language. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.

\n\n

In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming

\n\n

Bonus Rounds

\n\n

We also used a few weeks to read and discuss specific papers as a group.

\n\n

The first paper we read was Cardelli’s Typeful Programming. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.

\n\n

It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.

\n\n

Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.

\n\n

After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli’s language seem rather mundane. However, it’s likely that at the time he published it, these ideas were significantly newer and much less widespread.

\n\n

The other paper we read as a group was Alan Kay’s The Early History of Smalltalk. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.

\n\n

Final Thoughts

\n\n

Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!

\n\n

Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.

\n\n

Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.

")) ((? . 9) f post (u . "Continuations") (? . 9) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/07/17/continuations/index.html" . unix) (u . "/blog/2017/07/17/continuations/") (u . "2017-07-17T12:52:07") (? . 10) (? . 66) (c (u . "history") c (u . "Author: Ben Greenman")) (u . "\n

From the PRL archives:

\n\n
\n

It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. — Continuations by Alan Nall, Indiana University, 1983

") #t (u . "\n

From the PRL archives:

\n\n
\n

It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. — Continuations by Alan Nall, Indiana University, 1983

\n\n\n
\n\n

I first encountered this essay on continuations in a green folder in the PRL. It turns out, the author spent a semester at Indiana University working on the same fringe problem for a graduate-level programming languages course. According to the instructor: “What he said was true. He could not stop thinking about the problem the entire semester.” This essay was a kind of final exam.

")) ((? . 11) f post (u . "Lexical and Dynamic Scope") (? . 11) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/09/05/lexical-and-dynamic-scope/index.html" . unix) (u . "/blog/2019/09/05/lexical-and-dynamic-scope/") (u . "2019-09-05T10:00:00") (? . 32) (? . 38) (c (u . "scope") c (u . "definitions") c (u . "history") c (u . "Author: Ming-Ho Yee")) (u . "\n

This all started with a simple question about the R programming language: is R lexically or dynamically scoped?

\n\n

To answer that question, we need to understand what scope is, along with lexical scope and dynamic scope.

") #t (u . "\n

This all started with a simple question about the R programming language: is R lexically or dynamically scoped?

\n\n

To answer that question, we need to understand what scope is, along with lexical scope and dynamic scope.

\n\n\n

In this blog post, I’d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a subsequent post, I’ll discuss scoping in R and why it can be confusing.

\n\n

What is scope?

\n\n

Scope refers to the places in a program where a variable is visible and can be referenced.

\n\n

An interesting situation is when a function has free variables. Consider the example below:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7
\n
\n 
x <- 1\nf <- function(a) x + a\ng <- function() {\n  x <- 2\n  f(0)\n}\ng() # what does this return?\n
\n
\n
\n\n

On line 1, we create a mapping for x with value 1. On line 2, we define a function f whose body uses the parameter a, but also the free variable x. On line 3, we define a function g, whose body creates a new mapping for x with value 2, and then calls f(0). (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call g().

\n\n

What value does g return when it is called? What mapping does the free variable x on line 2 refer to? Does it refer to the mapping on line 1 that was visible when f was defined? Or does it refer to the mapping on line 4 that was created just before f was called?

\n\n

Lexical scoping

\n\n

Under lexical scoping (also known as static scoping), the scope of a variable is determined by the lexical (i.e., textual) structure of a program.

\n\n

In the example above, the definition of x on line 1 creates a scope that starts after its definition and extends into the bodies of f and g. However, the second definition of x on line 4 creates a new scope that (1) shadows the previous definition of x, and (2) does not extend into the call f(0) on line 5. Looking at this from another direction, the use of x on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.

\n\n

Therefore, under lexical scoping, the example program returns 1.

\n\n

Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which definition each variable refers to—but it may not be able to determine the values of each variable.

\n\n

Dynamic scoping

\n\n

Under dynamic scoping, a variable is bound to the most recent value assigned to that variable, i.e., the most recent assignment during the program’s execution.

\n\n

In the example above, the free variable x in the body of f is evaluated when f(0) is called on line 5. At that point (during program execution), the most recent assignment was on line 4.

\n\n

Therefore, under dynamic scoping, the example program returns 2.

\n\n

Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.

\n\n

(Edited 2020/08/13: As of Emacs 27.1, “lexical binding is now used by default when evaluating interactive Elisp.” Thanks to Artem Pelenitsyn for bringing this to my attention.)

\n\n

Now for a digression

\n\n

These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.

\n\n

However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with—a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.1

\n\n

I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for dynamic scope not mention scope at all? If scope is about “where a variable is visible,” and that definition is with respect to a variable definition, then why do so many explanations and examples define lexical and dynamic scope in terms of variable use?

\n\n

Scope and Extent

\n\n

I found some answers in Guy Steele’s Common Lisp the Language, 2nd Edition,2 which Matthias Felleisen recommended to me.

\n\n

In chapter 3, Steele introduces the concepts of scope and extent:

\n\n
\n

Scope refers to the spatial or textual region of the program within which references may occur. Extent refers to the interval of time during which references may occur.

\n\n

In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:

\n\n\n\n

Steele points out that dynamic scope is a misnomer, even though it is both a traditional and useful concept. It can be defined as indefinite scope and dynamic extent. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.

\n\n

Discussion

\n\n

I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of “dynamic scope”—it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!

\n\n

The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.

\n\n

A brief and incomplete history of lexical scope

\n\n

During my research of different definitions of lexical scope, I began to wonder if there was an “original” definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.

\n\n\n\n

Next stop, R

\n\n

Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. Is R lexically or dynamically scoped?

\n\n

In the next blog post, we’ll answer that question, and also see how R can be very confusing.

\n\n

I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.

\n\n
\n\n
\n
    \n
  1. \n

    For example, at one point I defined lexical/dynamic scoping in terms of a “lexical environment” and a “dynamic environment.” But (1) that’s a circular definition, (2) it assumes the reader has some intuition of how a “lexical environment” is different from a “dynamic environment,” and (3) it conflates two different kinds of “environment.” ↩

    2. \n
  2. \n

    G. Steele. “Scope and Extent,” in Common Lisp the Language, 2nd ed. 1990. [Available online] ↩

    3. \n
  3. \n

    J. McCarthy. “Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,” Communications of the ACM, vol. 3, no. 4, April 1960. [DOI][Available online] ↩

    4. \n
  4. \n

    J. McCarthy. “History of LISP,” in History of Programming Languages, 1978. [DOI][Available online] ↩

    5. \n
  5. \n

    P. Naur (ed.). “Revised Report on Algorithmic Language ALGOL 60,” Communications of the ACM, vol. 6, no. 1, January 1963. [DOI][Available online] ↩

    6. \n
  6. \n

    P. Landin. “The mechanical evaluation of expressions,” The Computer Journal, vol. 6, no. 4, January 1964. [DOI][Available online] ↩

    7. \n
  7. \n

    J. Moses. “The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,” SIGSAM Bulletin 15, July 1970. [DOI][Available online] ↩

    8. \n
  8. \n

    G. Sussman and G. Steele. “SCHEME: An Interpreter for Extended Lambda Calculus.” 1975. [Available online] ↩

    9. \n
  9. \n

    G. Steele and G. Sussman. “The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).” 1978. [Available online] ↩

")) ((? . 12) f post (u . "Categorical Semantics for Dynamically Typed Programming Languages") (? . 12) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/index.html" . unix) (u . "/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/") (u . "2017-05-01T12:25:17") (? . 87) (? . 3) (c (u . "HOPL") c (u . "category theory") c (u . "dynamic typing") c (u . "gradual typing") c (u . "Author: Max New")) (? . 5) #t (u . "\n\n

In 1969, Dana Scott wrote an unpublished manuscript in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote a paper that organized many of the different semantics he and others had since found using the language of category theory.

\n\n

This latter paper is really the first deserving of the title “categorical semantics of dynamic typing”, and so I’m going to present some of the theorems and “theorems” presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.

\n\n

My Full Notes continue the story, and you might also be interested in the discussion during the lecture.

")) ((? . 13) f post (u . "Beta Reduction (Part 1)") (? . 13) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/11/02/beta-reduction-part-1/index.html" . unix) (u . "/blog/2016/11/02/beta-reduction-part-1/") (u . "2016-11-02T21:10:18") (? . 57) (? . 41) (c (u . "lambda") c (u . "calculus") c (u . "beta") c (u . "reduction") c (u . "semantics") c (u . "Author: Milo Davis")) (u . "\n

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

") #t (u . "\n

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

\n\n\n

Introduction

\n\n

This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn’t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.

\n\n

This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in Racket, OCaml, and Haskell. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I’ll explain how to prove that β-reduction is confluent.

\n\n

The λ-Calculus

\n\n

The λ-calculus is a simple model of computation developed by Alonzo Church. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The BNF for the λ-calculus is as follows:

\n\n
e ::= x\n   | λx.e\n   | e e
\n\n

In the above BNF, x is a metavariable, standing for any variable. In this post, I use x, y, z, a, and b as variables in my examples. The λx.e term represents a function with a single parameter x. We say that the parameter, x is bound in e. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.

\n\n

If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn’t want to program in this language as it’s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I’m not going to discuss these ideas in detail, but if you’re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in Python, Scheme, and JavaScript.

\n\n

Now that we’ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I’m assuming the Barendregt variable convention (2.6) which states that every bound variable is distinct from every free variable.

\n\n
x[ x := e ] = e\ny[ x := e ] = y\n(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ])\n(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])
\n\n

With the substitution function defined, we can write a semantics for evaluation:

\n\n
------------------------------\n  (λx.e1) e2 ->β e1[ x := e2 ]\n\n    e1 ->β e1'\n--------------------\n  e1 e2 ->β e1' e2\n\n    e2 ->β e2'\n--------------------\n  e1 e2 ->β e1 e2'
\n\n

These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).

\n\n

What is β-reduction?

\n\n

More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function’s parameter in its body. More formally, we can define β-reduction as follows:

\n\n
(λx.e1) e2 = e1[ x := e2 ]
\n\n

Given that definition, lets look at a few examples. I’ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It’s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called “call by value”. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I’m going to ignore that here. This presentation of Haskell’s semantics is called “call by name” and the optimization is called “call by need”. (For more details on lazy evaluation, I recommend: Chang and Felleisen, ESOP 2012.)

\n\n

Some Examples

\n\n

The first example evaluates the same way in all of the languages.

\n\n
    (λx.x) (λy.y)\n->β x[ x := (λy.y) ]\n=   (λy.y)
\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
    ((λ (x) x) (λ (y) y))\n->β x[ x := (λ (y) y) ]\n=   (λ (y) y)\n
\n
\n
\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
    (fun x -> x) (fun y -> y)\n->β x[x := (fun y -> y) ]\n=   x[x := (fun y -> y) ]\n
\n
\n
\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
    (\\x -> x) (\\y -> y)\n->β x[ x := (\\y -> y) ]\n=   (\\y -> y)\n
\n
\n
\n\n

In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
    (\\x -> \\y -> y) ((\\z -> z) (\\a -> a))\n->β (\\y -> y)[x := ((\\z -> z) (\\a -> a))]\n= (\\y -> y)\n
\n
\n
\n\n

Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
    ((λ (x) (λ (y) y)) ((λ (z) z) (λ (a) a)))\n->β ((λ (x) (λ (y) y)) (z[ z := (λ (a) a) ]))\n=   ((λ (x) (λ (y) y)) (λ (a) a))\n->β (λ (y) y)[ x := (λ (a) a) ]\n=   (λ (y) y)\n
\n
\n
\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
    (fun x -> (fun y -> y)) ((fun z -> z) (fun a -> a))\n->β (fun x -> (fun y -> y)) (z[ z := (fun a -> a) ])\n=   (fun x -> (fun y -> y)) (fun a -> a)\n->β (fun y -> y)[ x := (fun a -> a) ]\n=   (fun y -> y)\n
\n
\n
\n\n

The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:

\n\n
    (λx.λy.y) ((λz.z) (λa.a))\n->β (λx.λy.y) (z[ z := (λa.a) ])\n=   (λx.λy.y) (λa.a)\n->β (λy.y)[ x := (λa.a) ]\n=   (λy.y)
\n\n
    (λx.λy.y) ((λz.z) (λa.a))\n->β (λy.y)[ x := ((λz.z) (λa.a)) ]\n=   (λy.y)
\n\n

Lets look at a final example. This one is more complicated than the previous ones. I’m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what’s going on.

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
    ((λ (x) x) (((λ (a) (λ (b) (a b))) (λ (y) y)) (λ (z) z)))\n->β ((λ (x) x) ((λ (b) ((λ (y) y) b)) (λ (z) z)))\n->β ((λ (x) x) ((λ (y) y) (λ (z) z)))\n->β ((λ (x) x) (λ (z) z))\n->β (λ (z) z)\n
\n
\n
\n\n

The same thing happens in OCaml

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
    (fun x -> x) ((fun a -> (fun b -> a b)) (fun y -> y) (fun z -> z));;\n->β (fun x -> x) ((fun b -> (fun y -> y) b) (fun z -> z));;\n->β (fun x -> x) ((fun y -> y) (fun z -> z))\n->β (fun x -> x) (fun z -> z)\n->β (fun z -> z)\n
\n
\n
\n\n

In Haskell, the situation is a little different:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
    (\\x -> x) ((\\a -> \\b -> a b) (\\y -> y) (\\z -> z))\n->β (\\a -> \\b -> a b) (\\y -> y) (\\z -> z)\n->β (\\b ->  (\\y -> y) b) (\\z -> z)\n->β (\\y -> y) (\\z -> z)\n->β (\\z -> z)\n
\n
\n
\n\n

Finally, in the λ-calculus, things can go a few different ways.

\n\n
    (λx.x) ((λa.λb.a b) (λy.y) (λz.z))\n->β (λx.x) ((λb.(λy.y) b) (λz.z))\n->β (λx.x) ((λy.y) (λz.z))\n->β (λy.y) (λz.z)\n->β (λz.z)
\n\n
    (λx.x) ((λa.λb.a b) (λy.y) (λz.z))\n->β (λa.λb.a b) (λy.y) (λz.z)\n->β (λb.(λy.y) b) (λz.z)\n->β (λy.y) (λz.z)\n->β (λz.z)
\n\n
    (λx.x) ((λa.λb.a b) (λy.y) (λz.z))\n->β (λx.x) ((λb.(λy.y) b) (λz.z))\n->β (λb.(λy.y) b) (λz.z)\n->β (λy.y) (λz.z)\n->β (λz.z)
\n\n

There’s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:

\n\n
(λx.λy.y) ((λa.a a) (λb.b b))
\n\n

A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don’t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I’ll explain the proofs that the λ-calculus does indeed have these properties.

\n\n

P.S. In Racket, you can look at the examples using the stepper which will allow you to interactively run the term and see how it reduces.

")) ((? . 14) f post (u . "PLISS: Oregon without the greek") (? . 14) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/02/28/pliss-oregon-without-the-greek/index.html" . unix) (u . "/blog/2017/02/28/pliss-oregon-without-the-greek/") (u . "2017-02-28T23:01:00") (? . 26) (? . 2) (c (u . "event") c (u . "lectures") c (u . "language implementation") c (u . "Author: Jan Vitek")) (u . "\n

What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or PLISS for short) has set out to answer.

") #t (u . "\n

What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or PLISS for short) has set out to answer.

\n\n\n

\"PLISS

\n\n

The school will feature twelve speakers versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.

\n\n

Lectures cover current research and future trends in programming language design and implementation, including:

\n\n\n\n

A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.

\n\n

Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.

\n\n

More information at:

\n\n

https://pliss2017.github.io

\n\n

(Oregon is a reference to the OPLSS, in which you may also be interested.)

")) ((? . 15) f post (u . "Top Five Results of the Past 50 Years of Programming Languages Research") (? . 15) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/04/04/top-five-results-of-the-past-50-years-of-programming-languages-research/index.html" . unix) (u . "/blog/2017/04/04/top-five-results-of-the-past-50-years-of-programming-languages-research/") (u . "2017-04-04T10:21:36") (? . 21) (? . 85) (c (u . "HOPL") c (u . "Author: Ben Greenman")) (u . "\n

Over the past 50 years, which result from programming languages research has had the greatest impact on working programmers?

") #t (u . "\n

Over the past 50 years, which result from programming languages research has had the greatest impact on working programmers?

\n\n\n

The center of the universe for a working programmer is the language (or languages) they use. Fundamental results in programming languages (PL) research can re-shape this universe.

\n\n

In HOPL two weeks ago, Matthias claimed that type soundness is the most useful and influential result to flow from PL research to PL practice in the last 50 years.

\n\n

But 50 years is a long time, and there are many serious contenders for the title of greatest PL result. Here are my (alphabetized) picks for the top five:

\n\n

Abstraction

\n\n
\n

My goal in library design is this; I want to have a precise, elegant, re-usable abstraction —Conal Eliott, BayHac 2014 (00:01:55)

\n\n

By abstraction, I mean anything whose interface is not just “read the implementation”. Could be a tuple, module, object, structure, semaphore, macro, etc. Even the memory hierarchy pyramid in your operating systems textbook is an abstraction. They are everywhere, and they are what separates computer science (it’s about ideas) from electrical engineering (it’s about transistors). Thank you Peter Landin and J.H. Morris.

\n\n

Generational Garbage Collection

\n\n

I don’t know much about garbage collection. I do know that I want it, and I’m pretty sure that I wouldn’t have it (outside of research languages) without generational garbage collection. Thank you David Ungar.

\n\n

Generic Programming

\n\n

a.k.a. the mainstream interpretations of parametric polymorphism

\n\n

The thought of programming in Java 1.4 is terrifying. Thank you Jean-Yves Girard and John C. Reynolds and Gilad Bracha and Martin Odersky and David Stoutamire and Philip Wadler.

\n\n

Modularization

\n\n

How can humans understand large software systems? By organizing the systems into smaller components (modules, objects) with well-defined interfaces. It’s hard to imagine, but once upon a time the question of how to divide a system into modules was a new research problem. Thank you D.L. Parnas.

\n\n

Type Soundness

\n\n

Let me make two modest claims:

\n\n\n\n

Neither of these claims were true 50 years ago. They are definitely true today. And the slogan “well typed programs do not go wrong (up to a well-defined set of runtime errors)” has become the catchphrase of PL research. Thank you Robin Milner.

\n\n

Honorable Mentions

\n\n\n\n
\n\n

If you liked this post, you may also be interested in:

\n\n")) ((? . 16) f post (u . "Final Algebra Semantics is Observational Equivalence") (? . 16) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/index.html" . unix) (u . "/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/") (u . "2017-09-27T15:44:57") (? . 18) (? . 17) (c (u . "category theory") c (u . "math") c (u . "final encoding") c (u . "observational equivalence") c (u . "Author: Max New")) (u . "\n

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

\n\n

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

") #t (u . "\n

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

\n\n

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

\n\n\n

Intuition: Implementing a Signature

\n\n

As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.

\n\n

We could specify such a datatype by specifying a little language of numbers and finite multisets. We’ll have two “sorts” num and multiset, a constant for every integer, and an addition function

\n\n
'n : () -> num;\nadd : (num, num) -> num
\n\n

subject to the silly-looking equation:

\n\n
add('n,'m) = '(n + m)
\n\n

and some operations on multisets

\n\n
empty : () -> multiset;\nsingleton : (num) -> multiset;\nunion : (multiset, multiset) -> multiset;\nremove : (num, multiset) -> multiset;\ncount : (num, multiset) -> num
\n\n

subject to the computational equations:

\n\n
count('n, empty) = '0\ncount('n, singleton('n)) = '1\ncount('n, singleton('m)) = '0\ncount('n, union(s,t)) = add(count('n,s), count('n, t))\ncount('n, remove('n,s)) = '0\ncount('n, remove('m,s)) = count('n,s)
\n\n

These are “all” of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively want for reasoning about our programs. For instance, clearly union should be commutative, and remove should be idempotent, but it’s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the “initial algebra”. In Haskell, we could say

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13
\n
\n 
data MultiSet = Empty \n  | Singleton Integer\n  | Union MultiSet MultiSet\n  | Remove Integer MultiSet\n  deriving (Eq)\n\ncount :: Integer -> MultiSet -> Integer\ncount n Empty = 0\ncount n (Singleton m) | n == m = 1\ncount n (Singleton m) | n /= m = 0\ncount n (Union s t) = (count n s) + (count n t)\ncount n (Remove m s) | n == m = 0\ncount n (Remove m s) | n /= m = count n s\n
\n
\n
\n\n

Then it is completely obvious that all of our equations hold, but then Union is not commutative, as ghci will tell us:

\n\n
\n \n \n \n \n
\n
\n 
1\n2
\n
\n 
> (Singleton 1 `Union` Singleton 2) == (Singleton 2 `Union` Singleton 1) \nFalse\n
\n
\n
\n\n

However, there is another encoding that will give us that union is commutative and remove n is idempotent and actually every equation we could possibly want! It’s called the “final encoding” or “final algebra”. In Haskell, this looks like:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24
\n
\n 
data MultiSet' = MultiSet' { _count :: Integer -> Integer }\n\ncount' :: Integer -> MultiSet' -> Integer\ncount' n m = _count m n\n\nempty :: MultiSet'\nempty = MultiSet' { _count = \\n -> 0 }\n\nsingleton :: Integer -> MultiSet'\nsingleton n = MultiSet' { _count = \\m -> if n == m\n                                         then 1\n                                         else 0 }\n\nunion :: MultiSet' -> MultiSet' -> MultiSet'\nunion s t = MultiSet' { _count = \\n -> (count' n s) + (count' n t) }\n\nremove :: Integer -> MultiSet' -> MultiSet'\nremove n s = MultiSet' { _count = \\m -> if n == m\n                                        then 0\n                                        else count' n s }\n\ntest' = and [ count' n s == count' n t | n <- [0..1000]]\ns = singleton 1 `union` singleton 2\nt = singleton 2 `union` singleton 1\n
\n
\n
\n\n

Now we can verify that union is commutative because

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
union s t = MultiSet' { _count = \\n -> (count' n s) + (count' n t) }\n          = MultiSet' { _count = \\n -> (count' n t) + (count' n s) }\n\t\t  = union t s\n
\n
\n
\n\n

since + is commutative. Equality isn’t decidable anymore so I can’t give you a simple piece of code to witness this, but we can test our example before and we won’t be able to distinguish them, no surprise:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4
\n
\n 
> let s = singleton 1 `union` singleton 2\n> let t = singleton 2 `union` singleton 1\n> and [ count' n s == count' n t | n <- [0..1000]]\nTrue\n
\n
\n
\n\n

How do we know this is the “best” or at least “most canonical” implementation of our datatype? The intuition is that we really don’t care at all how our multisets are implemented as long as they behave the right way with respect to count since count returns an Integer, a type we do understand. Our encoding accomplishes this by representing a multiset s by the partially applied function \\n -> count n s.

\n\n

The formal name for this idea is observational equivalence. We say that two closed terms s,t of sort multiset are observationally equivalent if for any term C of type num that has s as a subterm, we can swap t in for s and prove that the two terms are equal. For instance C might be count(3, union(s, singleton(3))) or add(4,remove(5,s)). Then we’ve reduced the possibly complicated equality for multiset to the simple equality of num.

\n\n

Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see here), but let’s see what all this talk about “algebras”, initial or final is all about.

\n\n

Formalization Attempt 1: Algebras of a Theory

\n\n

First, our little language of numbers and multisets is called a theory. The specific category gadget that we’ll use to describe it is a multi-sorted Lawvere theory, or just Lawvere theory for short.

\n\n

A Lawvere theory is a category with finite products all of whose objects are finite products of a collection of sorts \\(S\\). We can construct this category from our little language above by making the objects be contexts \\(x:num,y:multiset,...\\) and morphisms \\(\\Gamma \\to\nx_1:s_1,...,x_n:s_n\\) to be \\(n\\)-tuples of terms \\(\\Gamma \\vdash t_1 : s_1,...,\n\\Gamma \\vdash t_n : s_n\\) modulo the equations we’ve specified. We’ll use the letter \\(T\\) to mean a Lawvere theory.

\n\n

Then a \\(T\\)-algebra is a denotational semantics of our theory \\(T\\), i.e., a product preserving functor \\(A : T \\to Set\\). This means for every sort we get a set \\(A(s)\\) and for every term \\(x_1:s_1,...,x_n:s_n\n\\vdash t : s\\) a function \\(A(t) : A(s_1)\\times\\cdots \\times A(s_n) \\to\nA(s)\\).

\n\n

Finally a morphism of \\(T\\)-algebras from \\(A\\) to \\(B\\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \\(A\\) to \\(B\\), but concretely this means for every sort \\(s\\) we get a function \\(\\alpha_s : A(s) \\to B(s)\\) that translates \\(A\\)s interpretation of \\(s\\) as a set into \\(B\\)s. The key property that we want is that the operations according to \\(A\\) and \\(B\\) do the same thing as determined by \\(\\alpha\\). Specifically, for any term \\(x_1:s_1,...,x_n:s_n \\vdash t :\ns\\), and inputs \\(x_1 \\in A(s_1),...,x_n \\in A(s_n)\\) we should get the same result if we evaluate \\(A(t)(x_1,\\ldots,x_n)\\) and then apply \\(\\alpha_s\\) as if we first translate \\(x_1,\\ldots,x_n\\) to \\(B(s_1),\\ldots,B(s_n)\\) and then apply \\(B(t)\\). If you unwind the definitions, this is exactly what naturality says.

\n\n

Then we have a category we’ll call \\(T-Alg\\) of \\(T\\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them always exist.

\n\n

The initial algebra is most famous here, we define for each sort \\(In(T)(s) = \\cdot \\vdash s\\), the closed terms of that sort modulo the equivalence of the theory, and \\(In(T)(s_1,\\ldots,s_n) =\nIn(T)(s_1)\\times\\ldots,In(T)(s_n)\\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \\(T\\)-algebras from \\(In(T) \\to A\\) is, we see that we don’t have any choice, the only one is the one that maps \\(\\cdot \\vdash t : s\\) to \\(A(t)\\) and this makes all the right diagrams to commute. This is pretty similar to our definition of “initial algebra” before, except that this time we defined count as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for count.

\n\n

However, an egregious flaw presents itself when we look at what the final algebra is. It’s completely trivial! We can define \\(Fin(T)\\) to take every sort to a one element set \\(Fin(T)(s) = \\{*\\}\\) and every term to the trivial function \\(\\{*\\}^n \\to \\{*\\}\\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.

\n\n

Formalization: Algebras of a Theory Extension

\n\n

To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts num and multiset in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know exactly what numbers are!

\n\n

To formalize this we’ll call the full theory \\(T_1\\) and the theory with just numbers \\(T_0\\). Then there should be a map from \\(T_0\\) to \\(T_1\\) that is the inclusion of theories. We’ll formalize this as a morphism of theories. A morphism of theories is a strict product-preserving functor from one theory to another. The strictness ensures that we don’t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it’s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don’t want to consider all such morphisms, only the ones that are “conservative extensions”, which means

\n\n
    \n
  1. there are no new closed terms at old types
    2. \n
  2. closed terms that were different before remain different.
\n\n

In our example (1) ensures that we don’t add any new exotic numbers like undefined or ∞, and (2) ensures that we keep \\(0\\) different from \\(1\\), like the final algebra did before by having all numbers have the same interpreation \\(*\\).

\n\n

We can formalize this in the following way. Note that any morphism of Lawvere theories \\(m : T \\to S\\) induces a functor on the category of algebras \\(m^* : S-Alg \\to T-Alg\\) by just composing functors. An \\(S\\)-algebra is a functor from \\(S\\) to sets, and \\(m\\) is a functor from \\(T\\) to \\(S\\) so we can compose to get \\(m^*(A)(t) = A(m(t))\\).

\n\n

Now, we can express the idea of a conservative extension by saying that the canonical arrow from \\(In(T)\\) to \\(m^*(In(S))\\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \\(T\\) up to \\(T\\)-equivalence are isomorphic to the closed terms of the type provided by \\(m\\) in \\(S\\) up to \\(S\\)-equivalence. This is an equivalent formulation to the definition in Mitch’s paper, but there it is separated into two properties fullness and faithfulness, and doesn’t use the initial algebras and \\(m^*\\) explicitly.

\n\n

Now we can verify that the inclusion \\(i : T_0 \\to T_1\\) of the number theory into the number-multiset theory is an extension in this sense.

\n\n

Finally we can define our notion of \\(i\\)-algebra, which will be our correct notion of algebra. An \\(i\\)-algebra is a \\(T_1\\) algebra \\(A\\) such that

\n\n
    \n
  1. The canonical algebra map \\(! : In(T_0) \\to m^*A\\) is an isomorphism.
    2. \n
  2. The canonical algebra map \\(! : In(T_1) \\to A\\) is surjective i.e., for each sort \\(s, !_s\\) is surjective.
\n\n

The first condition says again that we have a conservative extension of \\(T_0\\), but the second is more interesting. It says that every denotation given by \\(A\\) is represented by some term in \\(T_1\\). In fact what it really ensures is that \\(A\\) determines a congruence relation on \\(T_1\\) given by \\(t1 \\equiv_A t2\\) if \\(A(t1) = A(t2)\\). In light of this, the first condition could be called adequacy.

\n\n

Furthermore, the surjectivity condition ensures that any morphism of \\(i\\) algebras, i.e., a map as \\(T_1\\)-algebras is also surjective, so a morphism \\(A \\to B\\) is a witness to the fact that \\(B\\) determines a stronger congruence relation on \\(T_1\\) than \\(A\\) does: \\(t1 \\equiv_B t2\n\\implies t1 \\equiv_A t2\\). Then asking for a final algebra is asking for exactly the:

\n\n
\n

Strongest adequate congruence relation

\n\n

which is exactly the definition of observational equivalence you will find in, say Pitt’s chapter of Advanced TAPL. There is a difference in the meaning of adequacy, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it’s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that adequacy is not a predetermined concept, we have to pick \\(T_0\\) and \\(i\\) in order to know what adequacy means.

\n\n

Conclusion: Tying it back to Final Encodings

\n\n

So now we’ve seen that

\n\n
\n

Final algebras are equivalent to initial algebras modulo observational equivalence

\n\n

Of course we haven’t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn’t use very many properties of sets in our definition, so it’s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.

\n\n

Another question is how to implement this when we have a proper type theory and not just some boring sorts. In particular, if we have function types, then we won’t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?

")) ((? . 18) f post (u . "PLT Redex FAQ") (? . 18) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/09/25/plt-redex-faq/index.html" . unix) (u . "/blog/2017/09/25/plt-redex-faq/") (u . "2017-09-25T23:39:16") (? . 74) (? . 16) (c (u . "tutorial") c (u . "PLT Redex") c (u . "Author: Ben Greenman") c (u . "Author: Sam Caldwell")) (u . "\n

A short guide to Redex concepts, conventions, and common mistakes.

") #t (u . "\n

A short guide to Redex concepts, conventions, and common mistakes.

\n\n\n
\n\n

To contribute to this FAQ, submit issues and pull requests to: https://github.com/nuprl/website/

\n\n

Q. What is Redex useful for?

\n\n
    \n
  1. declaring regular tree grammars
    2. \n
  2. defining pattern-based judgments and relations on terms
    3. \n
  3. testing properties of the above
\n\n

More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.

\n\n

Q. What is Redex not useful for?

\n\n

Proving theorems about a grammar, judgment, or relation.

\n\n

Q. What is a term?

\n\n

Informally, a term is:

\n\n\n\n

More formally, a term is the result of evaluating (term X), where X is any syntactically-correct Racket expression.

\n\n

Examples:

\n\n
$ racket\nWelcome to Racket v6.10.0.3.\n> (require redex/reduction-semantics)\n> (term 42)\n42\n> (term (+ 2 2))\n'(+ 2 2)\n> (term (\"hello\" world (#false)))\n'(\"hello\" world (#f))
\n\n

Some terms may look strange. That’s OK, because a term by itself has no meaning.

\n\n

Terms can refer to previously-defined values using the unquote escape.

\n\n
> (define x (term 42))\n> (term (+ 2 x))\n'(+ 2 x)\n> (term (+ ,x (unquote x)))\n'(+ 42 42)
\n\n

Q. What is a Redex model?

\n\n

A Redex model is collection of tools for working with terms. The tools may include:

\n\n\n\n

The goal of these tools is to encode a “real thing” (maybe, a programming language) using Redex terms.

\n\n

Q. What is a language?

\n\n

A Redex language is a named set of non-terminals, patterns, and binding forms. For example, a Redex model of the natural numbers might start with this language:

\n\n
(define-language nat\n  [N ::= Zero\n         (Plus1 N)])
\n\n\n\n

Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.

\n\n

The non-terminal N describes all terms that are either:

\n\n
    \n
  1. the symbol Zero
    2. \n
  2. lists of the form (Plus1 N), where N is either Zero or another “Plus1”
\n\n

For example,

\n\n
(term Zero)\n(term (Plus1 Zero))\n(term (Plus1 (Plus1 Zero)))\n(term (Plus1 (Plus1 (Plus1 Zero))))\n;; .... and so on
\n\n

If a language has binding forms, then some terms can introduce names. See the question on binding forms (below) for an example.

\n\n

Q. What is a pattern?

\n\n

A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains named non-terminals from the language, then you can ask whether a Redex term matches the pattern.

\n\n

A named non-terminal for a language L is an identifier made of: (1) a non-terminal from L, (2) an underscore (_), and (3) any other identifier. See the FAQ entry below.

\n\n

For example, (redex-match? L p t) returns #true if the term t matches the pattern p relative to the language L.

\n\n
(define-language nat\n  [N ::= Zero (Plus1 N)])\n\n(redex-match? nat N_some-name (term Zero))\n;; #true\n(redex-match? nat (Plus1 N_a) (term Zero))\n;; #false\n(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero))))\n;; #true
\n\n

If (redex-match? L p t) is #true, then (redex-match L p t) shows how named non-terminals in the pattern bind to subterms of t.

\n\n
(redex-match nat N_0 (term Zero))\n;; (list (match (list (bind 'N_0 'Zero))))\n(redex-match nat (Plus1 N_0) (term Zero))\n;; #f\n(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero))))\n;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))
\n\n

Q. What is a named non-terminal?

\n\n

A named non-terminal in a language L is an identifier of the form X_Y, where:

\n\n\n\n

The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.

\n\n
(define-language trees\n  [binary-tree ::= Leaf\n                   (Node binary-tree binary-tree)])\n\n(redex-match trees\n  (Node binary-tree_left binary-tree_right)\n  (term (Node Leaf (Node Leaf Leaf))))\n;; (list\n;;  (match\n;;   (list (bind 'binary-tree_left 'Leaf)\n;;         (bind 'binary-tree_right '(Node Leaf Leaf)))))
\n\n

Q. What else can patterns express?

\n\n

Redex patterns may contain special identifiers to guide pattern-matching. For instance:

\n\n\n\n

Examples:

\n\n
(redex-match? nat (Plus1 _) (term (Plus1 9)))\n;; #true\n(redex-match? nat (N_0 ...) (term ()))\n;; #true\n(redex-match? nat (N_0 ...) (term (Zero)))\n;; #true\n(redex-match nat (N_0 ...) (term (Zero Zero Zero)))\n;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))
\n\n

See the Redex reference for the full pattern language, including the named and unique non-terminals of the form X_!_Y.

\n\n

Q. What can patterns not express?

\n\n\n\n

Q. What is a judgment?

\n\n

A Redex judgment form defines a relation on terms. The relation is defined by a set of inference rules.

\n\n

Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:

\n\n\n\n

Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.

\n\n
(define-language nat\n  [N ::= Zero (Plus1 N)])\n\n(define-judgment-form nat\n  #:mode (N= I I)\n  #:contract (N= N N)\n  [\n   --- Zero=\n   (N= Zero Zero)]\n  [\n   (where (Plus1 N_0--) N_0)\n   (where (Plus1 N_1--) N_1)\n   (N= N_0-- N_1--)\n   --- Plus1=\n   (N= N_0 N_1)])
\n\n
    \n
  1. the language is nat; Redex uses the language to interpret patterns
    2. \n
  2. the mode is (N= I I); this means N= is the name of a judgment that expects two input terms (or, N= is a binary relation on terms)
    3. \n
  3. the contract is (N= N N); in other words, N= expects two terms that match the N non-terminal from the nat language
    4. \n
  4. there are two inference rules, named Zero= and Plus1=
    5. \n
  5. the Zero= rule states that (N= Zero Zero) always holds
    6. \n
  6. the Plus1= rule states that (N= N_0 N_1) holds if N_0 and N_1 are both Plus1 terms whose contents are related by N=
\n\n

The where clauses are guards. When Redex tries to apply a rule with premises of the form (where pattern term), it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See the Redex reference for more.

\n\n
(judgment-holds (N= Zero Zero))\n;; #true\n(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero))))\n;; #true\n(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero))))\n;; #false
\n\n

Note: the inference rules form a set, not a sequence. So when you ask Redex whether (judgment-holds (N= Zero Zero)), it applies all rules that match (N= Zero Zero). For N= this is just one rule, but in general it could be many rules.

\n\n

Q. What is a judgment form #:mode?

\n\n

A #:mode declaration expects a list of the form (id pos-use …), where id is an identifier and each pos-use is either I or O. These declarations say four things:

\n\n
    \n
  1. id is the name of a new judgment form
    2. \n
  2. id expects N arguments, where N is the number of pos-use symbols
    3. \n
  3. id expects an input at each position where the mode contains an I
    4. \n
  4. id produces an output at each position where the mode contains an O
\n\n

For example, a type inference judgment may take an expression as input and output a type. Here’s a fast and easy type inference judgment for arithmetic expressions. Given any term e_0, the judgment outputs the type Int.

\n\n
(define-language Arith\n  (e ::= integer (e + e))\n  (τ ::= Int))\n\n(define-judgment-form Arith\n  #:mode (infer-type I O)\n  #:contract (infer-type e τ)\n  [\n   --- T-Int\n   (infer-type e_0 Int)])
\n\n

Q. What can judgments not express?

\n\n

Redex does not support inference rules that require guessing.

\n\n

One example of this is a transitivity rule: \"τ_0 is related to τ_2 if there exists a τ_1 such that τ_0 is related to τ_1 and τ_1 is related to τ_2\". The following example tries to define a transitive subtyping (<:) relation, but Redex rejects the S-Trans rule.

\n\n
(define-language SomeTypes\n  (τ ::= (→ τ τ) Integer))\n\n(define-judgment-form SomeTypes\n  #:mode (<: I I)\n  #:contract (<: τ τ)\n  [\n   (<: τ_0 τ_1)\n   (<: τ_1 τ_2)\n   --- S-Trans\n   (<: τ_0 τ_2)]\n  [\n   --- S-Refl\n   (<: τ_0 τ_0)]\n  [\n   (<: τ_dom-1 τ_dom-0)\n   (<: τ_cod-0 τ_cod-1)\n   --- S-Arrow\n   (<: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])
\n\n

The error is that in the rule S-Trans, the named non-terminal τ_1 appears in an input position but is not bound to a term.

\n\n

Q. What is a metafunction?

\n\n

A metafunction is a term-level function on terms.

\n\n

Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.

\n\n

Here is a metafunction that returns #true when given two equal natural numbers. The definition is similar to the N= judgment form.

\n\n
(define-metafunction nat\n  N=? : N N -> boolean\n  [(N=? Zero Zero)\n   #true]\n  [(N=? N_0 N_1)\n   (N=? N_0-- N_1--)\n   (where (Plus1 N_0--) N_0)\n   (where (Plus1 N_1--) N_1)]\n  [(N=? N_0 N_1)\n   #false])
\n\n\n\n

Any occurrence of (N=? ….) in any term is evaluated.

\n\n
(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero))))\n;; #true\n(term ((N=? Zero Zero) Zero))\n;; '(#true Zero)\n(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero))))\n;; #false
\n\n

Any occurrence of N=? outside a term is an error.

\n\n

Metafunction where-clauses are analogous to judgment form where-clauses. See the Redex reference for more.

\n\n

Note: the cases in a metafunction form a sequence, not a set. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.

\n\n

Q. Should I use a metafunction or a judgment form?

\n\n

Use a judgment form.

\n\n

Metafunctions are handy, but judgments are easier to read and debug and maintain.

\n\n

Q. What is a reduction relation?

\n\n

A reduction relation is a set of term-rewriting rules.

\n\n

Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.

\n\n\n\n

See the Redex reference for a full description of the guards.

\n\n

The preferred way to define a reduction relation is to define a language that includes three non-terminals:

\n\n
    \n
  1. a non-terminal for the domain of the reduction relation
    2. \n
  2. a non-terminal for a subset of the domain that cannot reduce further
    3. \n
  3. a non-terminal for evaluation contexts
\n\n

An evaluation context is a term that contains a hole. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite — in particular, the sub-term that matches the hole.

\n\n

In the following example, bexp is the domain of a reduction relation. A bexp term represents a boolean expression, which can be #true or #false or a conjunction of expressions or a disjunction of expressions. The boolean expressions #true and #false are also values (val); these cannot reduce further. The non-terminal E is for evaluation contexts.

\n\n
(define-language Bool\n  (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp))\n  (val ::= #true #false)\n  (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E)))\n\n(define step\n  (reduction-relation Bool\n    #:domain bexp\n    [--> (in-hole E (val_lhs ∧ val_rhs))\n         (in-hole E val_new)\n         ∧-step\n         (where val_new ,(and (term val_lhs) (term val_rhs)))]\n    [--> (in-hole E (val_lhs ∨ val_rhs))\n         (in-hole E val_new)\n         ∨-step\n         (where val_new ,(or (term val_lhs) (term val_rhs)))]))
\n\n

The function apply-reduction-relation applies a reduction relation to a term and returns a list of ways that the term can step.

\n\n
(apply-reduction-relation step (term #true))\n;; '()\n(apply-reduction-relation step (term (#true ∧ #true)))\n;; '(#true)\n(apply-reduction-relation step (term (#true ∧ #false)))\n;; '(#false)\n(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true)))\n;; '((#true ∧ #true))
\n\n

Three things about the reduction relation step:

\n\n
    \n
  1. Using in-hole on the first argument of —> searches a term for a subterm that Redex can apply a reduction rule to.
    2. \n
  2. Using in-hole on the second argument of —> puts a new value back into the hole in the evaluation context.
    3. \n
  3. The unquote operator (,) escapes to “Racket mode” (see below) to evaluate a conjunction or disjunction.
\n\n

A judgment or metafunction is a formal alternative to “escaping to Racket”, but escaping can be convenient.

\n\n

Note: the cases in a reduction relation form a set, not a sequence. If more than one case matches, Redex applies them all.

\n\n

Q. What is “Racket mode”? What is “Redex mode”?

\n\n

Code in a Redex model is sometimes evaluated in “Racket mode” and sometimes evaluated in “Redex mode”. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.

\n\n

Key points:

\n\n
    \n
  1. A Redex program starts in Racket mode.
    2. \n
  2. The X in (term X) is evaluated in Redex mode …
    3. \n
  3. … unless X contains unquoted sub-expressions. Unquoting escapes to Racket mode …
    4. \n
  4. … and terms inside an unquoted sub-expression are evaluated in Redex mode.
\n\n

In other words, term enters Redex mode and unquote (,) escapes back to Racket.

\n\n

Redex implicitly switches to Redex mode in a few other places, for example:

\n\n\n\n

When in doubt, try using an unquote. Redex will raise an exception if it finds an unquote in Racket mode.

\n\n

Q. Are side-conditions evaluated in “Racket mode” or “Redex mode”?

\n\n

A (side-condition e) sometimes evaluates e as a Racket expression and sometimes evaluates e as a Redex expression.

\n\n\n\n

Q. What is a binding form?

\n\n

In the lambda calculus, λ-terms bind variables. A term (λ x M) means that any free occurrence of x in the sub-term M refers to the x from the λ-term.

\n\n

Redex can express this idea with a binding form.

\n\n
(define-language Λ\n  [e ::= (e e) x (λ x e)]\n  [x ::= variable-not-otherwise-mentioned]\n  #:binding-forms\n  (λ x_0 e_0 #:refers-to x_0))
\n\n

Note: all the non-terminals in a language must be defined before the #:binding-forms keyword. If a non-terminal definition appears after the #:binding-forms keyword, then Redex will interpret the “definition” as a binding form.

\n\n

Binding forms work together with Redex’s functions for substitution and alphabetic equivalence.

\n\n
(alpha-equivalent? Λ\n  (term (λ x x))\n  (term (λ y y))))\n;; #true\n\n(define-metafunction Λ\n  test-substitute : e -> e\n  [(test-substitute (λ x_0 e_0))\n   (substitute e_0 x_0 y)])\n(term (test-substitute (λ z (z z))))\n;; '(y y)
\n\n

Q. What is …? What is ….?

\n\n

Three dots (…) is for building patterns. If p is a pattern then (p …) matches any list whose elements all match p.

\n\n
(define-language L)\n(redex-match? L (number ... boolean ...) (term (1 2 #true #true)))\n;; #true
\n\n

Four dots (….) may be used in define-extended-language to extend a previously-defined non-terminal.

\n\n
(define-language C\n  (keyword ::= auto break case))\n(define-extended-language C++\n  C\n  (keyword ::= .... class))\n\n(redex-match? C keyword (term auto))\n;; #true\n(redex-match? C keyword (term class))\n;; #false\n(redex-match? C++ keyword (term auto))\n;; #true\n(redex-match? C++ keyword (term class))\n;; #true
\n\n

Q. Where to learn more about Redex?

\n\n

“Critical path” resources:

\n\n\n\n

“Procrastination” resources:

\n\n")) ((? . 19) f post (u . "[Making an IDE Plugin for DrRacket (cross-post)](https://lang.video/blog/2018/03/21/making-an-ide-plugin-for-drracket/)") (? . 19) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/index.html" . unix) (u . "/blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/") (u . "2018-04-12T12:12:53") (? . 6) (? . 37) (c (u . "tutorials") c (u . "Author: Leif Andersen")) (? . 5) #f (? . 5)) ((? . 20) f post (u . "[Ř Overview I (cross-post)](https://www.o1o.ch/lab/entry/1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc.html)") (? . 20) 1731622765 (p+ #"/home/runner/work/website/website/blog/2022/02/16/-r\314\214-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/index.html" . unix) (u . "/blog/2022/02/16/-r%CC%8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/") (u . "2022-02-16T00:00:42") (? . 0) #f (c (u . "Ř JIT") c (u . "Author: Olivier Flückiger")) (? . 5) #f (? . 5)) ((? . 21) f post (u . "[What even is compiler correctness? (cross-post)](https://williamjbowman.com/blog/2017/03/24/what-even-is-compiler-correctness/)") (? . 21) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/03/24/-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness/index.html" . unix) (u . "/blog/2017/03/24/-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness/") (u . "2017-03-24T23:11:17") (? . 82) (? . 15) (c (u . "Author: William J. Bowman")) (? . 5) #f (? . 5)) ((? . 22) f post (u . "Artifacts for Semantics") (? . 22) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/05/15/artifacts-for-semantics/index.html" . unix) (u . "/blog/2017/05/15/artifacts-for-semantics/") (u . "2017-05-15T10:08:31") (? . 23) (? . 51) (c (u . "Author: Daniel Patterson")) (u . "\n

Gabriel Scherer and I recently wrote an artifact for a semantics paper on a typed assembly language interoperating with a high-level functional language.

") #t (u . "\n

Gabriel Scherer and I recently wrote an artifact for a semantics paper on a typed assembly language interoperating with a high-level functional language.

\n\n\n

We wrote a interpreter, typechecker, and parser in OCaml, compiled it to Javascript using js_of_ocaml, and then put it on a webpage (with an editor with syntax highlighting and error reporting) that allows people to step through examples from the paper or write their own. (Feel free to start by playing a bit with our artifact).

\n\n

This post will summarize the different parts to make it easier for others to repeat this process. We think it was a total success, and have gotten feedback that it makes understanding the (somewhat complex) language from the paper much easier. We argue that building such interpreters / typecheckers is easy enough that all papers should do this. Further, while our interpreter / typechecker is completely unverified, since we wrote it in OCaml, this approach should work equally well for semantics verified in Coq and then extracted to OCaml.

\n\n
\n\n

The paper in question, FunTAL: Reasonably Mixing a Functional Language with Assembly (to appear in PLDI17), presents a multi-language that incorporates a typed assembly language (TAL) and a simple functional language where each can be embedded within the other. The paper then develops a logical relation that can be used to reason about the equivalence of such mixed programs. For example in the paper we show an imperative register-based factorial and a functional factorial equivalent.

\n\n

Both the static and dynamic semantics are relatively complex. The typed assembly has registers (which store word-sized values), a heap (which stores code-blocks and tuples), and a stack (not a call-stack, simply a place where word-sized values can be pushed and popped). Code-blocks have pre-conditions on the state of the registers and the stack, and allow the tail of the stack to be abstracted over polymorphically. This allows values to be protected on the stack before jumping to blocks that otherwise could change them. This is used, along with a novel notion of return markers, to ensure well-bracketing in the control flow of the typed assembly. The return markers indicate the location that points to the block that will eventually be returned to (assuming it doesn’t loop infinitely). At the top level, the return marker end indicates that, assuming it does not loop, eventually the program will stop, rather than returning somewhere else.

\n\n

Understanding the dynamic semantics requires tracking how values flow through the registers, the heap, and the stack, and rather than a call-stack, the user has to track the control flow through the statically-enforced return markers. This allows a good deal of low-level control-flow while still ensuring that calls will eventually return to the right place. This well-bracketing is vital to be able to reason about “components” that eventually return a value of a particular type, a necessity when embedding these components in a typed high-level program! However, it does mean that understanding the static and dynamic semantics from a few rules alone is a tall order. Our functional language is more standard, though we use (iso)-recursive types to allow recursion, which can easily trip up people, especially when you don’t have a type-checker to catch typos!

\n\n

For that reason, when working through examples for the paper I implemented a simple interpreter for the multi-language. I did this in OCaml, in the most straightforward way possible: by translating the definitions from the paper into type definitions (for F and for TAL), and the reduction relation into a “step” function that (assuming it wasn’t stuck or a value), did one step of evaluation. Later, I did the same thing for the type-checker, translating rules into a type-checking function. The latter had to deviate from the rules in the paper in a few minor ways, as the typing rules we had in the paper were slightly not syntax directed.

\n\n

Having the interpreter and type-checker was very useful for me, as I could check that the examples from the paper did not contain typos, but it was much less useful as an artifact for a reader of the paper. To use it the reader would have to download the source, install OCaml, write out examples as OCaml data constructors in a test file, compile it, run it, and then interpret the (quite overwhelming) output of every step of evaluation. At each step, I printed the current registers, current stack, current heap, what the evaluation context was (as you might be evaluating TAL instructions that were embedded inside a functional program that, in turn, was embedded in further TAL instructions), and what the current reduction was.

\n\n

To get from that useful-for-the-authors artifact to a useful-to-readers artifact requires doing three things:

\n\n
    \n
  1. Allow reading/writing programs in a notation as close to the paper as possible. In our paper we use superscripts, subscripts, and a few greek letters, but ended up with a syntax otherwise very close to the paper — the biggest differences were a few extra delimiters introduced to reduce ambiguity.
    2. \n
  2. Present an interface that highlights type errors at the location they occurred in, and allow a reader to step forward and backwards through the evaluation. Printing console output traces is fine for authors, but adds too much effort for readers.
    3. \n
  3. Put it online! Don’t require installing any software! Conveniently, implementing 2 is also made easier once done online, as we could use existing editor tooling to present the code, highlight errors, etc. By using OCaml, we were able to easily use the excellent js_of_ocaml.
\n\n

The first was done by Gabriel, who wrote a grammar using Menhir, and then equipped it with custom parsing error messages that provide much better feedback when there are typos in what people are trying. We also wrote a pretty-printer using the PPrint library, so we could show intermediate program states through the UI. After writing this, we were able to convert our existing suite of test cases and examples to be written textually, which was a huge improvement for us as well! These and other tests were used to ensure that the parser/pretty-printer would round-trip properly.

\n\n

For the interface, I built a simple web page that had the CodeMirror editor equipped with a very simple syntax highlighter (8 lines of code to highlight keywords & atoms, plus a CodeMirror extension to highlight matching brackets) and error highlighting (which is triggered by the OCaml code). I then made a simple “machine state” UI that showed, in pretty-printed format, the heap, stack, registers, context, and redex. On the OCaml side, when the “run” button is clicked, we parse and typecheck and, assuming no errors occur, store the current state as our “history”. As the user clicks forward or backwards, we run the step function and append to the history of states or pop states off of the history. In total, there are 50 lines of Javascript and about 150 lines of OCaml that handle the logic for this interactive UI.

\n\n

Putting it online was very easy, based on the choice of tools used earlier. We compile the main file (web.ml) to Javascript using js_of_ocaml, and it pulls in the parser, type-checker, interpreter, examples, etc. The rest of the artifact is a single html file, a CSS file, and a few javascript files for CodeMirror. It requires no server backend, is easy to archive and save, and will even run on smartphones!

\n\n

The total time spent implementing the artifact was a small fraction of the time spent on the paper (probably 15 days of person-time), and while it was not in any critical way essential for the success of the paper, it does make the paper much easier to read, and we would argue that all semantics papers would be better off with easy to use artifacts for experimentation. Also, while implementing the artifact we found a few mistakes in the typing judgments for the language. The most interesting one was for our protect TAL instruction, which exists to protect the tail of the stack in a fresh type variable. We had written this as a typing rule that type-checked the rest of the instruction sequence with the abstracted tail, but this never allowed the tail to be accessed again. By translating the typing judgments exactly into code, we realized that there was a problem, because examples that should have worked did not type-check! We were then able to fix the typing rule to conform to what we originally thought it achieved — locally abstracting, but not abstracting from outside the component. What is interesting is that this did not come up in our proofs because the typing rule was perfectly valid — it just did not allow non-trivial programs that used the protect instruction. It’s quite possible we would have noticed this without implementing the artifact, but the artifact certainly made it easier!

\n\n

To see the artifact online, visit:

\n\n

https://dbp.io/artifacts/funtal

\n\n

The source code is at:

\n\n

https://github.com/dbp/funtal

")) ((? . 24) f post (u . "Tutorial: Racket FFI, Part 2") (? . 24) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/06/29/tutorial-racket-ffi-part-2/index.html" . unix) (u . "/blog/2016/06/29/tutorial-racket-ffi-part-2/") (u . "2016-06-29T18:48:17") (? . 28) (? . 53) (c (u . "Racket") c (u . "FFI") c (u . "tutorial") c (u . "Author: Asumu Takikawa")) (u . "\n

This is part 2 of my tutorial on using the Racket FFI. If you haven’t read\npart 1 yet, you can find it\nhere.\nUpdate: part 3 is also now available\nhere.

\n\n

Part 2 will continue with more Cairo examples. In this installment, I plan to\ngo over some more advanced FFI hacking such as handling computed argument\nvalues, custom return arguments, and using C structs.

") #t (u . "\n

This is part 2 of my tutorial on using the Racket FFI. If you haven’t read\npart 1 yet, you can find it\nhere.\nUpdate: part 3 is also now available\nhere.

\n\n

Part 2 will continue with more Cairo examples. In this installment, I plan to\ngo over some more advanced FFI hacking such as handling computed argument\nvalues, custom return arguments, and using C structs.

\n\n\n

First, here’s the core code from part 1 condensed and re-arranged into an\nexample that you can copy and paste into your definitions area:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
#lang racket
(require racket/draw
         ffi/unsafe
         ffi/unsafe/define
         pict)
 
; bitmap magic
(define bt (make-bitmap 256 256))
(define bt-surface (send bt get-handle))
 
; C types
(define-cpointer-type _cairo_t)
(define-cpointer-type _cairo_surface_t)
(define _cairo_line_cap_t
  (_enum '(butt round square)))
 
(define cairo-lib (ffi-lib #f))
(define-ffi-definer define-cairo cairo-lib)
 
; the foreign functions
(define-cairo cairo-create
  (_fun _cairo_surface_t -> _cairo_t)
  #:c-id cairo_create)
(define-cairo cairo-move-to
  (_fun _cairo_t _double _double -> _void)
  #:c-id cairo_move_to)
(define-cairo cairo-line-to
  (_fun _cairo_t _double _double -> _void)
  #:c-id cairo_line_to)
(define-cairo cairo-set-line-width
  (_fun _cairo_t _double -> _void)
  #:c-id cairo_set_line_width)
(define-cairo cairo-stroke
  (_fun _cairo_t -> _void)
  #:c-id cairo_stroke)
(define-cairo cairo-set-line-cap
  (_fun _cairo_t _cairo_line_cap_t -> _void)
  #:c-id cairo_set_line_cap)
 
(define ctx (cairo-create bt-surface))
 
; Bitmap -> Pict
; a helper for displaying the bitmap
(define (show bt)
  (linewidth 2 (frame (bitmap bt))))
\n\n

Dashes and array arguments

\n\n

To start off, let’s look at another C example from the Cairo\nsamples page.\nThis time we will look at the \"dash\" example, which has a\nuse of an input array:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
double dashes[] = {50.0,  /* ink */
                   10.0,  /* skip */
                   10.0,  /* ink */
                   10.0   /* skip*/
                  };
int    ndash  = sizeof (dashes)/sizeof(dashes[0]);
double offset = -50.0;
 
cairo_set_dash (cr, dashes, ndash, offset);
cairo_set_line_width (cr, 10.0);
 
cairo_move_to (cr, 128.0, 25.6);
cairo_line_to (cr, 230.4, 230.4);
cairo_rel_line_to (cr, -102.4, 0.0);
cairo_curve_to (cr, 51.2, 230.4, 51.2, 128.0, 128.0, 128.0);
 
cairo_stroke (cr);
\n\n

The most interesting function here is cairo_set_dash, which takes an\narray argument. The only other new functions are cairo_rel_line_to\nand cairo_curve_to which have very straightforward C types:

\n\n
\n \n \n \n \n \n
\n \n \n \n \n \n \n \n
> (define-cairo cairo-rel-line-to
    (_fun _cairo_t _double _double -> _void)
    #:c-id cairo_rel_line_to)
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define-cairo cairo-curve-to
    (_fun _cairo_t
          _double _double
          _double _double
          _double _double
          -> _void)
    #:c-id cairo_curve_to)
\n\n

Meanwhile, the C type signature for cairo_set_dash from the Cairo\ndocs looks like this:

\n\n
\n \n \n \n \n \n \n \n \n \n
void cairo_set_dash (cairo_t *cr,
                     const double *dashes,
                     int num_dashes,
                     double offset);
\n\n

Something to note about the arguments is that num_dashes\nencodes the length of the array dashes. This will come up later when\nwe want to make the C type for this function more convenient.

\n\n

On the Racket side, it’s natural to represent the array of dashes as either a\nlist or vector of numbers. Given that, a fairly literal translation of\nthe C type above might look like the following:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define-cairo cairo-set-dash
    (_fun _cairo_t
          (_list i _double)
          _int
          _double
          -> _void)
    #:c-id cairo_set_dash)
\n\n

This type includes a type constructor we haven’t seen yet: _list.\nThis is a so-called custom function type that has special meaning\ninside of a _fun type. It lets you convert between a Racket list and\na C array. Since arrays are often used for both input and output of a C function,\nthe constructor requires you to specify the mode in which you are using the\narray.

\n\n

Since we only want to provide a list from the Racket side to C, we’ll use\nthe i input mode. We can then call the function like this:

\n\n
\n \n \n \n \n \n \n \n \n \n
(cairo-set-dash ctx
                (list 50.0 10.0 10.0 10.0)
                4
                -50.0)
\n\n

Note that because of how we defined the type of cairo-set-dash we had to\nprovide the length of the input list as a separate argument! This seems pretty\nsilly since it’s very easy to get the length of a Racket list and because this\nis a likely source of mistakes. It would be preferable to compute the length\nargument automatically.

\n\n

Luckily, the _fun type constructor actually lets you do this with the\n(name : type) syntax for naming arguments in combination with the\n(type = expr) syntax for supplying computed arguments:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define-cairo cairo-set-dash
    (_fun _cairo_t
          ; name this argument for later uses in the type
          [dashes : (_list i _double)]
          ; a computed argument position
          [_int = (length dashes)]
          _double
          -> _void)
    #:c-id cairo_set_dash)
\n\n

When a computed argument is specified with a =, it’s not necessary to provide\nthe argument on the Racket side. So cairo-set-dash is now an arity 3\nfunction that can be called like this:

\n\n
\n \n \n \n \n \n \n \n
(cairo-set-dash ctx
                (list 50.0 10.0 10.0 10.0)
                -50.0)
\n\n

This means we’ll never make a mistake in passing the length argument to Cairo.\nJust as an aside, it’s also possible to use Racket vectors instead of lists by using\nthe _vector type constructor.

\n\n

Putting it all together, we can reproduce the dashes example like this:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(define dashes '(50.0 10.0 10.0 10.0))
(define offset -50.0)
\n \n \n \n
(cairo-set-dash ctx dashes offset)
(cairo-set-line-width ctx 10.0)
\n \n \n \n
(cairo-move-to ctx 128.0 25.6)
(cairo-line-to ctx 230.4 230.4)
(cairo-rel-line-to ctx -102.4 0.0)
\n \n \n \n \n \n
(cairo-curve-to ctx 51.2 230.4 51.2
                    128.0 128.0 128.0)
\n \n \n \n
(cairo-stroke ctx)
\n \n \n \n
(show bt)
\n

\"image\"

\n\n

Result arguments and C structs

\n\n

For some more advanced FFI hacking, let’s consider the problem of drawing some\ntext into a predetermined space. In particular, we have our usual 256x256\nbitmap that we want to draw some text into:

\n\n
\n \n \n \n \n \n \n \n
> (define txt-bt (make-bitmap 256 256))
> (define txt-surface (send txt-bt get-handle))
> (define txt-ctx (cairo-create txt-surface))
\n\n

Our challenge is to make a Racket function that takes a string (let’s\nassume we can draw it in one line) and draws it into this bitmap.\nSince we are taking an arbitrary string, we will need to figure out\nhow to scale the text to fit. To make it simple, let’s just scale the\ntext to fit the width and assume the height will be okay.

\n\n

To implement the key step of measuring the text size, we can use the\ncairo_text_extents\nfunction. Its type signature is as follows:

\n\n
\n \n \n \n \n \n \n \n \n \n
void
cairo_text_extents (cairo_t *cr,
                    const char *utf8,
                    cairo_text_extents_t *extents);
\n\n

The interesting part of this signature is that\ncairo_text_extents_t\nis a struct type:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
/* from the Cairo docs */
typedef struct {
    double x_bearing;
    double y_bearing;
    double width;
    double height;
    double x_advance;
    double y_advance;
} cairo_text_extents_t;
\n\n

We haven’t yet seen how to handle C structs with the FFI, but it’s not\ntoo tricky. Support for C structs comes built-in and will look familiar\nif you’re used to Racket structs. We can directly translate the documented\ndefinition above into a define-cstruct declaration:

\n\n
\n \n \n \n \n \n
; the leading underscore is mandatory
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define-cstruct _cairo_text_extents_t
    ([x-bearing _double]
     [y-bearing _double]
     [width _double]
     [height _double]
     [x-advance _double]
     [y-advance _double]))
\n\n

This declaration does a couple of things. First, it defines a bunch of handy C types\nrelated to the struct for us:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> _cairo_text_extents_t
\n

#<ctype>

; pointer to struct
> _cairo_text_extents_t-pointer
\n

#<ctype>

; allows NULL pointer
> _cairo_text_extents_t-pointer/null
\n

#<ctype>

\n\n

Along with functions that look like regular Racket struct operations:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
; a struct constructor
> make-cairo_text_extents_t
\n

#<procedure:make-cairo_text_extents_t>

; a field selector
> cairo_text_extents_t-width
\n

#<procedure:cairo_text_extents_t-width>

; a predicate for the struct
> cairo_text_extents_t?
\n

#<procedure:^TYPE?>

; a field mutation function
> set-cairo_text_extents_t-width!
\n

#<procedure:set-cairo_text_extents_t-width!>

\n\n

With the struct type defined, it’s easy to come up with a rudimentary\ninterface for cairo-text-extents:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define-cairo cairo-text-extents
    (_fun _cairo_t
          _string
          _cairo_text_extents_t-pointer
          -> _void)
    #:c-id cairo_text_extents)
\n\n

In order to actually use this function, we need to create a text extents struct\nand provide it as a pointer. Conveniently, the FFI treats instances of C structs\nas pointers so this is pretty straightforward:

\n\n
\n \n \n \n \n \n \n \n \n \n
\n \n \n \n \n \n \n \n
> (define extents
    (make-cairo_text_extents_t
     0.0 0.0 0.0 0.0 0.0 0.0))
\n \n \n \n \n \n
> (cairo-text-extents
   txt-ctx \"hello world\" extents)
> (cairo_text_extents_t-width extents)
\n

54.0

\n\n

This style of programming feels awfully imperative though. Since we’re in a\nfunctional language, it would be nice to avoid the manual creation of the struct.\nWe can define an alternate version of the cairo-text-extents FFI wrapper\nby combining named arguments, a new _ptr type constructor,\nand a neat feature of _fun that lets you customize\nthe return result:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define-cairo cairo-text-extents*
    (_fun _cairo_t
          _string
          ; named args and _ptr
          [ext : (_ptr o _cairo_text_extents_t)]
          ; the return result of the C function
          -> _void
          ; custom return result for the wrapper
          -> ext)
    #:c-id cairo_text_extents)
\n\n

The _ptr constructor works like the _list constructor we saw\nearlier in this blog post but typically for a single object. Since we are passing\nin a value to use as an output, we specify the o mode to _ptr.\nIn output mode, this type will automatically allocate a new instance of the type\n(using the malloc function) and arrange for it to be passed in as\na pointer.

\n\n

The strangest part of this example is that there are now two uses of the\n-> form! By providing a second arrow, we can customize what the FFI wrapper\nreturns. The expression to the right of the second arrow is just Racket code that can\nreference previously named arguments. The result of evaluating this\nexpression is used instead of the normal return result for calls to the\nwrapped function. In this case, we just return the struct that was allocated for us.

\n\n

Using this new version of the wrapper is much simpler:

\n\n
\n \n \n \n \n \n
\n \n \n \n \n \n
> (cairo_text_extents_t-width
   (cairo-text-extents* txt-ctx \"hello world\"))
\n

54.0

\n\n

With that in hand, it’s pretty easy to write the function we set out to write:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(define-cairo cairo-show-text
  (_fun _cairo_t _string -> _void)
  #:c-id cairo_show_text)
 
(define-cairo cairo-scale
  (_fun _cairo_t _double _double -> _void)
  #:c-id cairo_scale)
 
; String -> Void
; draws a string scaled horizontally
(define (fit-text str)
  (define padding 20)
  (cairo-move-to txt-ctx (/ padding 2.0) 128.0)
  (define extents
    (cairo-text-extents* txt-ctx str))
  (define x-bearing
    (cairo_text_extents_t-x-bearing
     extents))
  (define width
    (cairo_text_extents_t-width
     extents))
  (define scale (/ (- 256.0 padding)
                   (+ x-bearing width)))
  (cairo-scale txt-ctx scale scale)
  (cairo-show-text txt-ctx str))
\n\n

And to conclude part 2 of this tutorial, here’s an example use\nof the new fit-text function:

\n\n
\n \n \n \n \n \n \n \n
> (fit-text \"Saluton, Mondo / Hallo, mundo\")
> (show txt-bt)
\n

\"image\"

")) ((? . 25) f post (u . "Bullets are good for your Coq proofs") (? . 25) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/02/21/bullets-are-good-for-your-coq-proofs/index.html" . unix) (u . "/blog/2017/02/21/bullets-are-good-for-your-coq-proofs/") (u . "2017-02-21T19:04:28") (? . 30) (? . 26) (c (u . "coq") c (u . "Author: Gabriel Scherer")) (u . "\n

I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.

\n\n

Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.

") #t (u . "\n

I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.

\n\n

Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.

\n\n\n

Why bullets

\n\n

While you are doing a proof, Coq shows a list of subgoals that have to be proved before the whole proof is complete. Most proof steps will operate on the current active subgoal, changing the hypotheses or the goal to prove, but some proof steps will split it into several subgoals (growing the total list of goals), or may terminate the proof of the current subgoal and show you the next active subgoal.

\n\n

Before bullets, a typical proof script would contain the proofs of each subgoal, one after another.

\n\n
induction foo. (* this creates many subgoal *)\n\nproof of first subgoal.\n\nproof of second subgoal.
\n\n

There are many ways to structure this to make the structure more apparent: people would typically have a comment on each subgoal, or make disciplined use of indentation and blank lines. But, in my experience, a major problem with this style was maintainability in the face of changes to the definitions or parts of automation. It could be very hard of what was happening when a proof suddenly broke after a change before in the file:

\n\n\n\n

What we need for robustness is a way to indicate our intent to Coq, when we think that a subgoal is finished and that a new subgoal starts, so that Coq can fail loudly at the moment where it notices that this intent does not match reality, instead of at an arbitrary later time.

\n\n

(The S*Case tactics used in (older versions of) Software Foundations can solve this problem if used in a carefully, systematic way, and additionally provides naming. Alexandre Pilkiewicz implemented an even more powerful cases plugin. Bullets are available in standard Coq since 8.4 (released in 2012), and can be used with no effort.)

\n\n

There is not much discussion of bullets around; see the documentation in the Coq manual. I learned a lot from Arthur Azevedo de Amorim’s Bullets.v file.

\n\n

Finally, some people don’t use bullets, because they systematically use so much automation that they never see subgoals — each lemma has a one-line proof. This is also a valid style. (I have been going to Adam Chlipala’s Formal Reasoning about Programs 2017 class, where Adam ignores bullets because that is his usual style.) Because I am not crushy enough to do this from the start, my proofs tend to start with cases and subgoals, and then I refine them to add more automation for robustness. I found bullets very useful for the first step, and during the refinement process.

\n\n

Bullets

\n\n

Bullets are actually a combination of two features, braces { ... } and actual list bullets — -, +, *, or homogeneous repetitions of those, for example -- or ***.

\n\n

Braces

\n\n

The opening brace { focuses the proof on the current subgoal. If you finish the proof of the subgoal, the following subgoal will not become accessible automatically; you have to use the closing brace } first. (If you finish the goal earlier than you think, you get an error.) Conversely, } fails if the subgoal is not complete. (If you fail to finish, you get an error.)

\n\n

The previous example can thus be written as follows, and will be more robust:

\n\n
induction foo. (* this creates many subgoal *)\n{\n  proof of first subgoal.\n}\n{\n  proof of second subgoal.\n}
\n\n

If you also want to make sure that an error occurs if the number of subgoals changes (for example if new constructors are added to the inductive type of foo), you can use an outer layer of braces:

\n\n
{ induction foo. (* this creates many subgoal *)\n  {\n    proof of first subgoal.\n  }\n  {\n    proof of second subgoal.\n  }\n} (* would fail if a new subgoal appeared *)
\n\n

List bullets

\n\n

A bullet, for example --, also focuses on the next subgoal. The difference is that when the subgoal is finished, you do not have a closing construction, you must use the same bullet to move to the next subgoal. (Again, this fails if the first proof step changes to prove too much or too little.) With bullets you would write

\n\n
induction foo. (* this creates many subgoal *)\n+ proof of first subgoal.\n+ proof of second subgoal.
\n\n

Bullets can be nested, but you must use different bullets for the different nesting levels. For example, if this proof is only one subgoal of a larger proof, you can use:

\n\n
- induction foo. (* this creates many subgoal *)\n  + proof of first subgoal.\n  + proof of second subgoal.\n- (* would fail if a new subgoal appeared *)\n  rest of the proof
\n\n

The natural ordering of bullets, I think, is by increasing number of lines: -, + then * (and then multi-character bullets, I guess). You can also mix bullets with braces: the opening brace resets the bullet scope, any bullet can be used again with the subgoal.

\n\n

This gives a large space of freedom in how you want to use these features. You can use only braces, only bullets, braces and only one level of bullets, etc. My own style evolved with experience using the feature, and I will present the current status below.

\n\n

My current bullet style

\n\n

When deciding how to use bullets, one distinguishes the commands that preserve the number of subgoals and those that may create new subgoals. I use some additional distinctions.

\n\n

Some tactics, for example assert, create a number of subgoals that is statically known, always the same for the tactic. I then use braces around each sub-proof, except the last one, which I think of as the “rest” of the current proof.

\n\n
assert foo as H.\n{ proof of foo. }\nrest of the proof using H:foo.
\n\n

(If the proof of foo takes several lines, I two-indent them, with the braces alone on their lines.)

\n\n

Most tactics create a dynamic number of subgoals, that depends on the specifics of the objects being operated on; this is the case of case, destruct, induction for example. In this case, I open a brace before the tactic, and use a bullet for each subgoal.

\n\n
{ induction foo; simpl; auto.\n- proof of first remaining subgoal.\n- proof of second remaining subgoal.\n  rest of the proof of the second subgoal.\n}
\n\n

(Notice that the subgoal-creating step is vertically aligned with the proof steps: I use both braces and bullets, but take only one indentation level each time.)

\n\n

As an exception, I may omit the braces if we are at the toplevel of the proof (Proof .. Qed serve as braces).

\n\n

Note that omitting the braces here and using different bullets when you nest is also just fine. In my experience it gives proofs that are a bit more pleasant to read but also a bit more cumbersome to edit and move around.

\n\n

Finally, a not-uncommon mode of use of “dynamic” tactics in the sense above is to expect all the cases, except one, to be discharged by direct automation (for example they are all absurd except one). When it is my intent that all cases but one be discharged (and not a coincidence), I express it by not using braces (this command preserves the number of subgoals), but marking the remaining subgoal with a new bullet without increasing the indentation level.

\n\n
{ induction foo.\n- first subgoal.\n- second subgoal.\n  case blah; discharge all sub-subgoals but one.\n+ remaining sub-subgoal of the second subgoal.\n  finish the sub-subgoal.\n- third subgoal.\n}
\n\n

(This is the only time where I use several bullet levels.)

\n\n

If you are the kind of programmer that is passionate about indentation style, I should now have tricked you to use bullets to propose a different variant. Otherwise, please consider using bullets anyway, for example by following the style above, it will make your life easier in the face of changing developments.

")) ((? . 27) f post (u . "Does anyone still care about printed proceedings? (Grab some at NEU this week!)") (? . 27) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/index.html" . unix) (u . "/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/") (u . "2016-06-13T10:50:14") (? . 1) (? . 28) (c (u . "proceedings") c (u . "dawn of the digital era") c (u . "Author: Gabriel Scherer")) (u . "\n

Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!

") #t (u . "\n

Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!

\n\n\n

If you are in the area and are interested, feel free to send me an email and come grab them. We have ASPLOS from XIII to XX, PLDI from 2005 to 2015 (but not 2014), and OOPSLA from 2002 to 2015. When I saw the stack, I grabbed the POPL ones from 2002 to 2015, but in fact I have no idea what to do with them and I’m a bit skeptical they would be of use to me; if you know you would use them, I would be glad to let you have them.

\n\n

If you were to buy those proceedings at conference-subscription rates today, it would cost you a small fortune. Yet nobody seems to want them. An odd disconnect, that I found amusing and maybe worthy of a blog post.

\n\n

But don’t get me wrong, the future of printed proceedings is not an important question. We should rather be asking ourselves: why are the products of the work of our communities not easily accessible in an Open Access long-term archive? Are you submitting your articles to arXiv, or another archive? Why not?

\n\n

Not caring about printed proceedings is perfectly fine; but please care about people outside institutions that want to access your work — for example, master student myself.

\n\n
\n\n

Update (August 2017): Gabriel returned to France and left the POPL proceedings at his desk. The proceedings are sitting in the hallway at NEU, in case anyone cares.

")) ((? . 29) f post (u . "Conversational Context and Concurrency") (? . 29) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/02/15/conversational-context-and-concurrency/index.html" . unix) (u . "/blog/2017/02/15/conversational-context-and-concurrency/") (u . "2017-02-15T01:21:55") (? . 62) (? . 30) (c (u . "HOPL") c (u . "Author: Tony Garnock-Jones")) (? . 5) #t (u . "\n\n

When programs are written with concurrency in mind, the programmer reasons about the interactions between concurrent components or agents in the program. This includes exchange of information, as well as management of resources, handling of partial failure, collective decision-making and so on.

\n\n

These components might be objects, or threads, or processes, or actors, or some more nebulous and loosely-defined concept; a group of callbacks, perhaps. The programmer has the notion of an agent in their mind, which translates into some representation of that agent in the program.

\n\n

We think about the contexts (because there can be more than one) in which agents exist in two different ways. From each agent’s perspective, the important thing to think about is the boundary between the agent and everything else in the system. But from the system perspective, we often think about conversations between agents, whether it’s just two having an exchange, or a whole group collaborating on some task. Agents in a conversation play different roles, join and leave the group, and build shared conversational state.

\n\n

In this talk, I used the idea of these conversational contexts as a lens through which to view the development of various metaphors and mechanisms of communication and coordination. I presented four computational models for concurrent interaction:

\n\n\n\n

These aren’t full programming languages, but there are many programming models that build upon them. In some cases, development of these ideas has progressed all the way up to system models including user interaction and so forth.

\n\n

The linked lecture notes include informal sketches of reduction semantics for each of the four models, plus a handful of small examples to give a feel for them.

\n\n

Lecture Notes:

\n\n\n\n

Discussion summary:

\n\n")) ((? . 10) f post (u . "Quotes and Stories from \"Turing 50\"") (? . 10) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/06/24/quotes-and-stories-from-turing-50/index.html" . unix) (u . "/blog/2017/06/24/quotes-and-stories-from-turing-50/") (u . "2017-06-24T20:00:52") (? . 8) (? . 9) (c (u . "dear diary") c (u . "Author: Ben Greenman")) (u . "\n

The ACM recently hosted a celebration of 50 years of the A.M. Turing award. These are some notes and thoughts from the event, including how Fred Brooks once rented a bus, Don Knuth’s outrageous implementation of batch processing, and Judea Pearl’s theory of homo sapiens.

") #t (u . "\n

The ACM recently hosted a celebration of 50 years of the A.M. Turing award. These are some notes and thoughts from the event, including how Fred Brooks once rented a bus, Don Knuth’s outrageous implementation of batch processing, and Judea Pearl’s theory of homo sapiens.

\n\n\n\n\n

Conventions / Disclaimers:

\n\n\n\n

Opening Remarks

\n\n

Alan Turing is with us today

\n\n

At the start of the event, the emcee unveiled a bronze bust of Alan Turing. This statue was on display at center stage during the whole event.

\n\n

It’s a good sculpture and it’s good we remember Alan Turing, but I’m sad that the ACM would encourage this kind of idol-worship. Let’s not forget Turing’s excellent teachers and colleagues!

\n\n

talk: Impact of Turing Recipients’ Work

\n\n
Barbara Liskov
\n\n
\n

the first awards recognized achievements in the standard fields of theory, AI, and systems

\n

hostile environment around the first awards, trepidation about future awards

\n

with Unix, Ritchie and Thompson got the design right

\n

Niklaus Wirth: “If I understood how important Pcode was, I would have spent more time designing it”

\n\n

thoughts

\n\n

What is “systems” — does that even have a definition? And Unix is definitely NOT an example of a “right design”; rather it’s a landmark of worse is better design.

\n\n

panel: Advances in Deep Neural Networks

\n\n

Stuart Russell

\n\n
\n

I work in all areas of AI except for deep learning

\n\n

Judea Pearl

\n\n
\n

I am a foreigner in this field … left because human beings are not good at handling information … people are very good with causal inference, not with statistical inference … deep learning is statistical

\n

there is a very old existence proof, homo sapiens took over the planet … I believe because they had an internal model of their environment … a drawing of a lion with wings is evidence of this model, you have to have such a model before you can experiment with it and imagine … snakes have superb optics, result of a long evolution process … very specific but they cannot build eyeglasses … humans have an internal model, can build a market based on promises and build large communities based on promises

\n

I see four levels … second level is predicting events, if I do X then what? … third level is counterfactual, if I did things differently then how would the outcome change … very hard to advance between levels, are we working to help machine learning ‘level up’?

\n

data science is about the relation between data and reality … data alone is not data science

\n\n
Michael Jordan
\n\n
\n

today we can’t think without holding a piece of metal

\n

machine learning is part of computer science rather than AI … AI is about how to make human … machine learning is about allocating resources … matrices are not all of human intelligence … neural nets are part of a wider toolbox … too much hype in NLP its just syntax

\n

huge gap between syntax and semantics … chat bots are just syntax, don’t learn … faking intelligence with neural nets, so well that you can build a company …

\n

real metric is task completion

\n

if I say ‘a GLEEB walked across the airport’ then true intelligence can make a lot of educated guesses about a ‘GLEEB’ without any other context

\n\n
Fei-Fei Li
\n\n
\n

I disagree, ML is part of AI … understanding intelligence and making intelligent methods for solving AI problems

\n

to quote Churchhill ‘its not beginning of end, not end, not beginning of end, probably end of beginning’

\n

todays AI powered by hardware and data

\n

AI cannot yet find our keys

\n

quote: ‘todays AI is making a perfect chess move while the world is on fire’ … ignores context

\n\n
Stuart Russell
\n\n
\n

Turing … a program is a mathematical object … math community did not recognize this

\n

lots of grad student descent … tuning to get performance … deep learning is neglecting the problem of exponential data … deep learning is just circuits, circuits lack expressive power … a human can process data from CERN but a neural net cannot, need to know physics

\n

probabilistic programming, somewhat under the radar, maybe on the right track … 10-line program running/generating/updating a large network of possibilities … more composable and flexible

\n\n
Ilya Sutskever
\n\n
\n

why I like deep learning … philosophically satisfying … the hypothesis class is a circuit … powerful hypothesis class not too many parameters … can actually find circuits … ‘violates all theory’ … really amazing … humans can see and hear pretty fast, even though our neurons are pretty slow, perhaps because we do a massively parallel process that doesn’t take many steps … works well enough to be useful

\n

models e.g. for vision are very hard to understand … fight fire with fire … incomprehensible solution to incomprehensible problem

\n\n
Raquel Urtasun
\n\n
\n

the breakthrough in neural nets is not algorithms … it is tricks, hardware, and grad students

\n

with neural nets we forget about modeling, uncertainty, and prior knowledge … perception is a canonical example

\n\n

question: boundaries

\n\n
Judea Pearl:
\n\n
\n

glad to see people in deep learning understand its limitations … is there a clearer definition of the boundaries? Are you worried about bridging the levels factual/inferential/counterfactural?

\n\n
Michael Jordan
\n\n
\n

the big problem is decision making under uncertainty

\n\n
Fei-Fei Li
\n\n
\n

cognition is a hard problem

\n\n
Judea Pearl
\n\n
\n

do you have a clear idea of the boundaries?

\n\n
Michael Jordan
\n\n
\n

neural nets use back-propagation … its non-modular, sad fact … performance and explainability is the tradeoff … then again people are non-modular

\n\n
Stuart Russell
\n\n
\n

AlphaGo is not deep learning … basically an improved version of the machines Arthur Samuel made in the late 1950s … the interesting code is in C++ … rules of go, next moves, searching future states … depends on transitive closure

\n\n
Judea Pearl
\n\n
\n

can AlphaGo take advice from a human?

\n\n
Michael Jordan
\n\n
\n

not currently, but that would be a new policy to add to the toolbox … just as neural nets are one tool within AlphaGo

\n\n
Raquel Urtasun
\n\n
\n

no reason to ask if deep learning is going to solve all problems

\n\n

question: education?

\n\n
Judea Pearl
\n\n
\n

indeed, what DO you teach in your neural networks classes?

\n\n
Fei-Fei Li
\n\n
\n

… chain rule, Taylor expansion

\n\n
Judea Pearl
\n\n
\n

teaching is communicating truths … what is true about neural nets? what are some things that will definitely not happen?

\n\n
Stuart Russell
\n\n
\n

Peter Norvig and I have a problem with our AI book … chapter on vision, chapter on speech, will probably post just point to the neural nets chapter … we don’t really understand! … really selling students short

\n\n
Fei-Fei Li
\n\n
\n

in labs we talk about what we cannot do … we all have open problems

\n

Stuart I hope you have a very good author for the chapters. There are so many open problems to communicate to students!

\n\n
Michael Jordan
\n\n
\n

CS cirriculum needs more statistics, inferential thinking … revise the whole cirriculum bottom-up to weave this in

\n\n

question: could a neural net fix my phone without breaking it?

\n\n
Judea Pearl
\n\n
\n

right! big problem that neural nets have no internal model to manipulate

\n\n

question: generalizability?

\n\n
Ilya Sutskever
\n\n
\n

special-purpose vs. general purpose solution depends on the problem … most things we give special-purpose solutions … I guess if you wanted to automate a mathematician that would need to be general

\n\n
Stuart Russell
\n\n
\n

always argue with your self … try to break what you’ve built … there’s a system that plays video games just using the pixels on screen as hints … it’s very good at mazes; if a newborn baby learned to play maze games in 2 hours that would be amazing! … does the system scale? absolutely not

\n\n

thoughts

\n\n

When Michael Jordan said “people are non-modular”, I think he means that people are able to break abstraction barriers when needed.

\n\n

panel: Restoring Personal Privacy without Compromising National Security

\n\n
Joan Feigenbaum
\n\n
\n

… wikileaks … russian hackers … social emergency …

\n\n
Whitfield Diffie
\n\n
\n

everything I say today is copyleft

\n

its a misunderstanding to talk about a conflict between security and privacy … two aspects … problem goes back to feudalism … the right to build a castle was granted by the king … on one hand a castle improves national security … on the other hand a castle can be used to attack the king … technology is upsetting the basic notion of private vs. public security … governments cannot protect citizens and cannot protect themselves … extremely difficult to prove that a small process is secure

\n

exceptional access makes it more complex

\n\n
Paul Syverson
\n\n
\n

major concern are national security threats and ability of authorities to confound threats … analogy to printing press … proclimation of 1635 that only state messengers can carry letters … 1663 treatise by the national censor, no printing house can have a back door … the general topic is very old … title of this session isn’t very good, the real dilemma is investigation vs privacy

\n\n
Bryan Ford
\n\n
\n

code is law for better or worse, tech is not a tool like a watch … tech can monitor us and decide when it works … tech is government, not obedient tools … the mind is a warrant-proof space … 5th amendment rights should extend to wearables

\n\n
Nadia Heninger
\n\n
\n

cannot divorce the security/privacy issues from the current political context … the serious vulnerabilities are not in math … they are in users and implementors

\n\n

question: back doors

\n\n
Joan Feigenbaum
\n\n
\n

perhaps we should explain what a back door is

\n\n
Bryan Ford
\n\n
\n

agency keeps a master key in escrow

\n

non-lawyers can and should take a stand on basic issues

\n

there are legitimate warrant-proof spaces … electronic extensions of the mind need to be recognized as warrant-proof spaces

\n

the set of authorities with backdoor access should change as I travel between countries … but this will lead to a global race to the bottom

\n\n
Whitfield Diffie
\n\n
\n

germany has a law against sex tourism (committed by German citizens visiting other countries) … neither government will be willing to lose backdoor access

\n\n
Nadia Heninger
\n\n
\n

technical reasons against backdoors … (1) ‘weak crypto’ was implemented, nobody turned it off, is now breakable by anyone in 2015 … (2) Juniper used non-default crypto parameters, someone (inside?) changed the parameters … (3) attackers exploit back doors

\n\n
Paul Syverson
\n\n
\n

quote ‘you can put a man on the moon, surely you can put a man on the sun’

\n\n
Whitfield Diffie
\n\n
\n

trouble is getting him back safely

\n\n
Bryan Ford
\n\n
\n

I think back doors are okay, but not for personal devices … need public lab and transparent processes, need separation of powers … prosecutors are getting cases thrown out because courts do not accept their backdoors … there is a place for transparent back door tools

\n\n
Nadia Heninger
\n\n
\n

politicians are rarely technical people

\n\n
Bryan Ford
\n\n
\n

tech is not a set of policy-neutral tools, need to address gap of understanding

\n\n

question: ???

\n\n
Whitfield Diffie
\n\n
\n

we don’t know how to build good crypto programs … opponents are debugging our programs with different goals … we’re trying for-all-paths safety (universal) … they’re trying exists-bad-path (existential)

\n\n
Bryan Ford
\n\n
\n

cybersecurity market is a lemon market

\n\n

question: how to advise

\n\n
Joan Feigenbaum
\n\n
\n

question from audience ‘I am an advisor to a company working with nuclear energy, they are terrified of being attacked, how should I advise them?’

\n\n
Whitfield Diffie
\n\n
\n

a network like that is probably separated enough to be safe … the problem is being safe AND connected to the web

\n\n
Bryan Ford
\n\n
\n

because the internet of things

\n\n

question: what should the ACM do?

\n\n
Nadia Heninger
\n\n
\n

maybe we need increased regulation, the ACM could help bring experts together

\n\n

question: what is true security

\n\n
Paul Syverson
\n\n
\n

it’s all the same thing … gets labeled differently … just trying to control which bits can go where and who gets to read them

\n\n
Nadia Heninger
\n\n
\n

security is the absense of being violated

\n\n
Paul Syverson: no true > security, need to consider context
\n\n
Joan Feigenbaum
\n\n
\n

problem of our community, have strict standards, may be unrealistic … maybe a lot more tolerance in practice than our model accepts

\n\n
Paul Syverson
\n\n
\n

security and privacy are environmental problems

\n\n

question: can we stop the needle-in-haystack search for vulnerabilities?

\n\n
Paul Syverson
\n\n
\n

need to build in security from the start

\n\n
Bryan Ford
\n\n
\n

need rule of law, transparency, separation of powers

\n\n
Whitfield Diffie
\n\n
\n

stop delaying, instead of spending $$$ on fixing problems, we should invest in solving the fundamental issues

\n\n

panel: Preserving our Past for the Future

\n\n

Note: I was volunteering during this session; quotes are sparse

\n\n
Mahadev Satyanarayanan
\n\n
\n

the running system is the total documentation … there are too many details for prose to capture

\n\n
??
\n\n
\n

running old code has a danger of running old bugs

\n\n
??:
\n\n
\n

what not to save? … it’s very hard to tell in advance

\n\n
Mahadev Satyanarayanan:
\n\n
\n

there is no absolute censor in a world with caching

\n\n
Brewster Kahle
\n\n
\n

asking UNESCO to solve the problem is unrealistic … need to empower the fanatics, given them tools to preserve data

\n\n

thoughts

\n\n

I totally agree with the “empower the fanatics” sentiment. Today, because of “volunteer librarians”, I think we’re doing pretty well about preserving the past. Suppose I found an old PowerPoint file. I’m sure I could find a way to read it with help from the internet — either by searching Google, pirating an old version of PowerPoint, or asking online forums. So personally I’m not worried about losing data we have currently; I’m more worried about the future, the internet becoming “less chaotic”.

\n\n

The panel raised a good question about how to preserve research and encourage reproducibility. A .pdf or .tex document is not enough; a virtual machine is okay. Really I think we need a stronger cultural emphasis on literate programming and a mature library like TeX to help authors store and share their work. The Gamma seems on the right track.

\n\n

I was surprised that the panel did not discuss search, version control, and the ACM’s open-access policy.

\n\n

panel: Moore’s Law is Really Dead: What’s Next?

\n\n

Note: I was volunteering during this session

\n\n
Butler Lampson
\n\n
\n

there’s plenty of room at the top … with Moore’s Law we got improvements at the bottom of the software stack, everything above got to benefit and it was easy to integrate the changes … there’s lots of opportunities to trim fat in the middle/top of the software stack … these improvements will be harder to integrate, but there’s lots of opportunities

\n\n
Margaret Martonosi
\n\n
\n

By the way, don’t believe the brochure that says I’m at Google. My affiliation is Princeton, Google and I are just friends.

\n\n
Butler Lampson
\n\n
\n

important to distinguish approximate vs. precise software … precise software has a specification and the customer cares about that specification … approximate software doesn’t have a hard spec, just needs to approximately work … the web is approximate, it doesn’t work and it doesn’t need to! … windows is precise, definitely has a spec and users definitely care

\n\n

thoughts

\n\n

The recording of this panel should be good; it was very lively, very practical. And the first audience question (by David Patterson) was “an A+ question”.

\n\n

The panel reminded me of a talk by Yale Patt about “the end” of the Von Neumann architecture. His opinion is future computers will be Von Neumann machines that rely on “accelerators” like a GPU — computer organization is not going to change, but will expand to have a bigger toolbox. So sure, Moore’s Law is dead, but there are many opportunities to make computers faster at places other than the bottom of the software stack.

\n\n

panel: Challenges in Ethics and Computing

\n\n

Note: I was volunteering during this session

\n\n
Raj Reddy
\n\n
\n

there are more slaves in the world currently than there were in the US during the civil war … here is one way technology could help, by giving everone a device to record their location … if someone’s time and location is constant, they may be held against their will

\n\n
??
\n\n
\n

do you believe every problem has a technological solution?

\n\n
Noel Sharkey
\n\n
\n

yes the training set may be biased against people similar to me, but I want you to consider my case as an individual

\n\n
??
\n\n
\n

a very nice Washington Post article

\n\n
??
\n\n
\n

whether to encrypt the back hall

\n\n
Raj Reddy
\n\n
\n

we can sit here and wring our hands, but nothing will come of it unless it is written in the US constitution

\n\n

thoughts

\n\n

I did not enjoy this panel. This is an ACM event, not a United Nations event. An ACM-sponsored panel about social and political problems should look for constructive ways that computer science can address these problems. Raj Reddy tried to give constructive solutions, but the panel seemed more interested in complaining about how hopeless things are.

\n\n

The comment by Noel Sharkey about “consider me as an individual” was something I hadn’t thought about. Instead of worrying about biased datasets, let’s use technology to collect data on an individual instead of abstracting a person by their race, age, or neighborhood.

\n\n

talk: Computer Science as a Major Body of Accumulated Knowledge

\n\n
Donald Knuth
\n\n
\n

don’t applaud me, just read my books

\n

at the time, computer science was AI, numerical analysis, and programming languages

\n

a colleague said ‘I will believe that computer science is a science when it has 1000 deep theorems’ … I am not sure what a deep theorem is but I think its different from what’s proven by deep learning

\n

great privilege that we can invent the problems we work on … imagination … physicists can only guess the size of the sun

\n

I’ve always believed computer science and math are two parallel subjects … sometimes you hear people wondering if one subsumes the other

\n

when I won the Turing Award, the prize money was about $1,000,000 less than it is today … I did get a nice Tiffany bowl that my wife and I use to serve strawberries … strawberries actually taste better …

\n

very fortunate in this field … I’m completely worthless as an economic advisor … it’s a game I’ve been able to take advantage of

\n\n

question: how could you offer to pay for TeX bug reports?

\n\n
Donald Knuth
\n\n
\n

well there were many, many bugs … I stopped doubling at 32768 … brought people out of nowhere … next time I check the bug reports will be 2021 … someone is queueing the bugs reports … I believe strongly in batch rather than swap-in/swap-out … last time I checked reports was 2014 so 2021 will be next

\n

TeX was a literate program, and it helped that I wrote ‘The Errors of TeX’ about the first N bugs

\n\n

question: do you think computers will become good composers of music? do you see a role for computer-assisted proving?

\n\n
Donald Knuth
\n\n
\n

Yes in part, assisted is the key word … I have a program running now that I hope will help me prove a theorem

\n\n

question: favorite algorithm?

\n\n
Donald Knuth
\n\n
\n

Tarjan’s strong components … short deep useful

\n\n

question: thoughts on AI, computers taking over?

\n\n
Donald Knuth
\n\n
\n

I get scared when I see Stuart Russell making assumptions based on people acting rationally … then you look at election results

\n\n

question: if you could start over and do things differently, what would you change?

\n\n
Donald Knuth
\n\n
\n

I would use decimal internally in TeX instead of binary

\n\n

question: how to record history?

\n\n
Donald Knuth
\n\n
\n

a video ‘Lets not dumb down the history of CS’ … used to be history of algorithms … trouble with funding … the history is nothing that a non-CS person could not understand … the whole field of history changed from internal to external … historians need to be external to get published in journals … no CS department supports a historian … recently read a dissertation about the ALGOL 60 copmiler … very careful, describes data structures and organization … this kind of thing is what deserves to be called history

\n\n

question: teachers

\n\n
Donald Knuth
\n\n
\n

hardest thing for me is choosing between two hypotheses (1) could teach this to anyone (2) only 2% of the world is geeks … suppose the second is true then you can’t talk about how to teach if the teacher is not in the 2% …

\n

the newest issue of CACM has a fun typo, ‘deep earning’

\n\n

panel: Quantum Computing: Far Away? Around the Corner? Or Maybe Both at the Same Time?

\n\n
John Martinis
\n\n
\n

goal to have a 45–50 qbit machine … 1 error per 1000 operations … to test, run sample algorithm, chart output vs. a classical supercomputer … got to be a supercomputer to finish the computation in time

\n\n
Andrew Yao
\n\n
\n

I’m a believer … one suggested benchmark is to factor 1000-digit numbers … impossible to attain … need to expore new possibilities, take physics attitute

\n

CS did well attracting attention to quantum … science should be more open … share results between physics chemistry CS … don’t just stick to your specialized conferences

\n

CS departments reception to quantum is less than satisfactory … 15 years ago, maybe 4 or 5 universities … now, maybe 7 or 8 .. China doing much better in this regard

\n\n
Jay Gambetta
\n\n
\n

not useful to make analogy to anything classical … universal fault tolerance? or computation in the presence of error … either would be excellent, still a long way off

\n

IBM put quantum on the cloud … picked an instruction set that tries to abstract away … have been 19 published papers on the behavior of this quantum hardware

\n\n
Dorit Aharonov
\n\n
\n

two paths … finding algorithms, besides Shor’s algorithm … make quantum computer to realize the algorithms … finding algorithms is very difficult … information-processing point-of-view

\n

error correction still small scale … can we use entanglement between probes to improve accuracy?

\n\n
Umesh Vazirani
\n\n
\n

_different goals … maybe you want perfect Qbits for a perfect Hilbert space … reality is a noisy space … short run, how to compute with noise … how to correct errors …

\n\n
Jay Gambetta:
\n\n
\n

_those 2 paths are the same to me … we want larger devices with fidelity

\n

lets build hardware see where goes … exciting prospect, computer scientists will explore what they can do with these erroneous qbits … that’s why IBM has the instruction set open to the community

\n\n

question: why isn’t adding 10 qbits only 10x harder?

\n\n
John Martinis:
\n\n
\n

building infrastructure to scale … not just grad student code … we’re all good coders using standard industry practices for coding

\n\n
Jay Gambetta
\n\n
\n

fidelity is hard to achieve

\n\n

question: both IBM and Google use superconducting storage?

\n\n
John Martinis
\n\n
\n

superconducting scales … ion traps harder to scale, but we still watch, keep eye on data

\n\n

question: education

\n\n
John Martinis
\n\n
\n

I like talking to engineering colleges … physics and engineering need to work together

\n\n

question: is quantum going to change programing languages?

\n\n
Jay Gambetta
\n\n
\n

yes very different to handle errors … current challenge is building an abstraction over the superconducting hardware

\n\n
John Martinis
\n\n
\n

hoping to first expose hardware, then get a model, eventually a language

\n\n
Dorit Aharonov
\n\n
\n

need to start with more algorithms

\n\n

question: what would Feynman do?

\n\n
John Martinis
\n\n
\n

experiments!

\n\n
Dorit Aharonov
\n\n
\n

yes he’d tell us to keep playing, and play with us

\n\n

panel: Augmented Reality: From Gaming to Cognitive Aids and Beyond

\n\n

Peter Lee starts off wearing a headset.

\n\n
Ivan Sutherland:
\n\n
\n

I can tell you how VR started. Bell Helicopter company wanted to land at night … put an infrared camera on the landing site and a display in the cockpit … to test they used the roof of their building … one day an observer in a Bell office is watching, though the camera, two men playing catch on the roof … one player threw the ball at the camera and the observer ducked … he had identified his position with the camera … my observation was that you didn’t need a camera, could substitute a computer … the rest is history

\n\n
Yvonne Rogers
\n\n
\n

my goal is to augment people … Englebart very inspiring … ok 2 stories … (1) a student of mine wanted to help picky eaters … computer vision for when they tried to hide peas under the plate … projected colors onto the peas, called them ‘disco peas’, kids were distracted enough to get over their aversion … children and parents got involved, new social interaction … (2) opera makeup for schoolchildren, virtually getting into character … teenage boys in the classes got to try makeup for the first time … singers found it useful for rehearsals

\n\n
Peter Lee
\n\n
\n

I feel socially awkward wearing this headset, but I have some of my slides here … making a wearable headset requires huge effort … research prototypes can be uncomfortable … a product needs to be perfect and its very hard to do perfect … one goal, give Lowe’s VR to demo a virtual kitchen … Case Western anatomy class used virtual cadaver, great collective experience

\n\n
Fred Brooks
\n\n
\n

two stories … (1) Henry Fuchs 1998, working with breast surgeon, try augmented reality to improve the precision of biopsy probe insertion … 2 years to a working prototype, hard to track surgeon’s eyes, display where probe is, where ultrasound is, provide low latency … one day trying on live patient, worked 100% perfect probe right on the mark, jubilation … then the doctor had to tell the patient ‘yes it is really cancer’ … (2) a challenge, augmented reality EMT training … real teams, virtual patient, virtual surround … track real tools, 8 eyes, 8 images, team needs to interact

\n\n

question: what are current uses of augmented reality?

\n\n
Ivan Sutherland
\n\n
\n

the pilot of a jumbo jet typically has 1 hour flight experience before he flies for the first time, but extensive training in a flight simulator

\n\n
Fred Brooks
\n\n
\n

the best AR

\n\n
Ivan Sutherland
\n\n
\n

once I was in a flight simulator with the chief pilot … and he turned to me and asked ‘have you ever experienced a slow roll in a 747?’ … a slow roll is a twisting motion, a very benign maneuver, constant one-G pressure the plane doesn’t know its upside down … ‘here we go’ and suddenly the world inverted … I remarked that it was certainly impressive, but didn’t you treat the simulator as a real experience, and never attempted anything you would not do in reality? … ‘that is true, but I am the chief pilot’

\n\n
Peter Lee
\n\n
\n

construction, architecture

\n\n
Fred Brooks
\n\n
\n

where’s the ‘augmented’?

\n\n
Peter Lee
\n\n
\n

whether augmented or virtual

\n\n
Fred Brooks
\n\n
\n

yes we did my kitchen that way, made my wife sick when she tried it

\n\n
Peter Lee
\n\n
\n

surgical

\n\n
Fred Brooks
\n\n
\n

still sounds virtual

\n\n
Yvonne Rogers
\n\n
\n

displays on a car, superimposed directions on the tarmac … one of the users took a vacation and had to use the old GPS technology … found it very difficult to go back

\n\n

question: AR tools for developers?

\n\n
Blair MacIntyre
\n\n
\n

can developers write apps for the Microsoft Hololens?

\n\n
Peter Lee
\n\n
\n

we belive in experience, anything we can do to foster experiences is good

\n\n
Fred Brooks
\n\n
\n

faking things … subtle and important … I remember using a flight simulator, navigating the runway, and I turned my head to see if my wing was going to clip a plane … turned and there was nothing there … emotional shock to leave the simulation, I had been flying for 1 hour

\n\n
Ivan Sutherland
\n\n
\n

pilot training is an early adopter because the cost of real planes is so high, impossible to train for emergency situations

\n

the ultimate goal, you can sit in a virtual chair … and if the chair has handcuffs you cannot get up … a virtual bullet is lethal … probably impossible because bits don’t weigh anything … you know Ben Franklin invented augmented reality, eyeglasses … the desire outweighs cost … I cannot see the audience here, maybe it would be good if I had a headset! but Peter took his off

\n\n
Peter Lee
\n\n
\n

because of my slides I couldn’t see the audience, but then without the headset I couldn’t see them either

\n\n

question: any challenges to suggest to the audience?

\n\n
Peter Lee
\n\n
\n

if we had holographic transport, we wouldn’t need planes!

\n\n
Yvonne Rogers
\n\n
\n

maybe, but you need to give attendees a physical presence … smell, touch

\n\n
Ivan Sutherland
\n\n
\n

what makes us willing to work together? I had a collaboration with three people … all in different locations .. communicated with a phone … worked perfectly, because we had worked in the same location first and knew one another so well … how to get to that point, where a simulation could be a useful tool … another good observation by Fred Brooks, given a domain X ask how good does the simulation need to be for X … Licklider told me, you’d need damn good fiction to land someone on the moon, the simulation would need to provide every detail … for flight simulation the user’s imagination can fill some gaps, a pilot can recognize an aircraft carrier from a rougher picture

\n\n
Fred Brooks
\n\n
\n

at IBM I once hired buses to bring the Poughkeepsie secretaries to the main office … the secretaries at the two offices only knew one another from the phone … this one lunch did so much good … only $75 to rent a bus

\n\n
Peter Lee
\n\n
\n

how important is it to shake hands, to bump into a table?

\n\n
Yvonne Rogers
\n\n
\n

for this conference, I think the live stream is getting a better experience because the cameras zoom in on us, the panelists … the audience in the back cannot see us, only a picture of us on the monitors

\n\n
Peter Lee
\n\n
\n

_one excellent video game, starts in the dark, you hear a voice … turn around and there’s a character sitting on a chair … if you rearrange your furniture he finds a new seat …

\n\n
Yvonne Rogers
\n\n
\n

games are a great example … Pokemon Go … Apple jusr released an app toolkit … need to get those in schools, in the hands of kids who can build with them

\n\n

question: Ivan, about your ‘ultimate display’ paper, what has since surprised or frustrated you?

\n\n
Ivan Sutherland
\n\n
\n

I wasn’t surprised because I never had any expectations … of course sticks are not real … no assumptions so no strong feelings

\n\n

question: people already distracted by cell phones, how to manage all this input?

\n\n
Yvonne Rogers
\n\n
\n

good question, how much data you can present to people … and then the problem with google glass, your companions don’t know what you are looking at … at least with snapchat glasses, you can trust the device is simpler

\n\n
Peter Lee
\n\n
\n

good writing defines reality, bad writing reports it … with the printing press, quickly went from 30,000 books to over 13,000,000 … novels evolved shortly after, a new form of expression

\n\n

question: Peter, how long do your people wear the hololens?

\n\n
Peter Lee
\n\n
\n

hard to say … but often longer than the battery lasts

\n\n
Fred Brooks
\n\n
\n

how long does it last?

\n\n
Peter Lee
\n\n
\n

depends what you’re doing, 3 hours

\n\n
Fred Brooks
\n\n
\n

that’s encouraging, we had a 30-minute cutoff because participants had enough

\n\n

question: nausea

\n\n
Peter Lee
\n\n
\n

I get nauseous in our minecraft VR … but there’s a pop-out feature where you keep playing, but the game world is in a TV set instead of around you … can pop back in when you’re feeling better

\n\n
Yvonne Rogers
\n\n
\n

we’ve seen about 20% of the population gets nauseous

\n\n
audience member
\n\n
\n

Dana Boyd conducted an experiment, found the nausea was worse for wemon

\n\n
Yvonne Rogers
\n\n
\n

oculus makes me feel sick, but the hololens has never given me trouble

\n\n
Peter Lee
\n\n
\n

have models to predict head motion, to keep the VR world steadier

\n\n
Blair MacIntyre
\n\n
\n

I remember reading papers that measured framerate … would be interesting to revisit

\n\n
Fred Brooks
\n\n
\n

framerate not important, its the latency that gets you … one colleague of mine, we call her ‘the canary’ because she’s so sensitive, in fact …

\n\n
Peter Lee
\n\n
\n

talking about nausea is part of the problem, people experience it more … every time I talk about it in public my co-workers tell me to stop!

\n

another cool application, there’s a hololens app to give blind people a tour of the Redmond office … you say a building and it takes you there

\n\n
Fred Brooks
\n\n
\n

one challenge, the relative brightness of the real and virtual worlds

\n\n

question: any last remarks

\n\n
Ivan Sutherland
\n\n
\n

_I hoped from the beginning that AR would be a teaching tool … I learned that F = MA not from a book but from a large flywheel in the school’s basement … very substantial inertia … the greatest value for AR would be to show people things in a way that makes the underlying meaning clear … what color should the hydrogen atoms in a benzene ring be? the color will be fiction, but the quality of learning will depend on that fiction … challenge for content makers … what is the haptic experience of feeling bits?

\n\n

Small Group Session

\n\n

After the last panel, I got to attend a small group session with other students, Dick Karp, and Don Knuth. It doesn’t feel right to summarize or quote from the session, but there’s one thing I want to write about.

\n\n

During the group session, I said something that I now regret. There was a brief silence as the group changed subjects, and someone suggested that we do a round of introductions. I objected, this will take so long, but in fact the introductions were a very good idea.

\n\n

Normally, I don’t like introductions because they focus on names, backgrounds, and credentials. I don’t care about any of these when I’m meeting someone! Rather, I prefer to just talk and by-the-way learn about the other person(s). There’s an anaology to double-blind reviewing — the focus should be content and not credentials.

\n\n

These introductions were successful for two reasons. First, they gave everyone in the room a turn to speak, and this seemed to help people join the actual discussion sooner. That was strange to me. I always feel a little nervous the first time I speak up in front of a group, but if I really feel like speaking then I can always get over this little barrier. I guess it’s not right to assume the nervousness is “little” for everyone. Second, the introductions format was “say your name and a funny fact”. This prompt by itself led to some nice conversation topics:

\n\n\n\n

“Nice” in the sense that everyone could contribute, which was a real challenge. Even the question “does anyone have a favorite algorithm?” didn’t have much success fostering discussion.

\n\n

Related: a useful greeting at the event was “what SIG are you?”. The answer was a good hint about what level of abstraction you two could best communicate at.

\n\n\n\n\n")) ((? . 31) f post (u . "Forgetful and Heedful contracts") (? . 31) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/04/07/forgetful-and-heedful-contracts/index.html" . unix) (u . "/blog/2019/04/07/forgetful-and-heedful-contracts/") (u . "2019-04-07T23:15:11") (? . 39) (? . 32) (c (u . "migratory typing") c (u . "higher-order contracts") c (u . "Author: Ben Greenman")) (u . "\n

Forgetful and heedful are two methods for space-efficient contracts developed by Michael Greenberg in 2014. These methods were born in the shadow of a third method, eidetic, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.

") #t (u . "\n

Forgetful and heedful are two methods for space-efficient contracts developed by Michael Greenberg in 2014. These methods were born in the shadow of a third method, eidetic, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.

\n\n\n
\n\n

Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of three parties: the API code, the client code, and the contract between them.

\n\n

For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:

\n\n
  #lang racket/base\n\n  (module sort racket\n    (provide\n      (contract-out\n        [quicksort\n          (-> (vectorof point/c) void?)]))\n\n    (define point/c (vectorof integer?))\n\n    (define (quicksort points)\n      ....))\n\n  (module client racket\n    (require (submod \"..\" sort))\n    (quicksort '()))\n\n  (require 'client)
\n\n
quicksort: contract violation;\n expected a vector\n  given: '()\n  in: the 1st argument of\n      (-> (vectorof (vectorof integer?)) void?)\n  contract from: \n      (file.rkt sort)\n  blaming: (file.rkt client)\n   (assuming the contract is correct)
\n\n

That covers the basics. For an extended introduction to contracts, visit The Racket Guide.

\n\n

The quicksort example and the related figures are from the paper Collapsible Contracts: Fixing a Pathology of Gradual Typing

\n\n

Classic contracts and “Space Efficiency”

\n\n

The (vectorof point/c) contract used above describes a possibly-mutable array whose elements match the point/c contract. Since the array can be mutated, this contract has implications for two parties:

\n\n
    \n
  1. the client module must supply a good array, and
    2. \n
  2. the sorting module must not insert a bad element.
\n\n

To enforce the second condition, the vectorof contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:

\n\n
(quicksort (vector (vector 4 4)\n                   (vector 2 2)\n                   (vector 1 1)\n                   (vector 3 3)))
\n\n

After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a solid blue bar for the write checks against the sort module and a striped blue bar for the read checks against the client.

\n\n

\"A

\n\n

In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted — because every write and read applies a contract to the element.

\n\n

\"Layers

\n\n

On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.

\n\n

Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and “indirecting” future writes/reads through wrappers. These space and time costs can add up.

\n\n
\n

“on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times” — Collapsible Contracts

\n\n

To fix the problem, researchers have been exploring space-efficient implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and eidetic, forgetful, and heedful are his names for three implementations.

\n\n

(Although the goal of this post is to promote forgetful and heedful, we will review all three.)

\n\n

Eidetic space-efficiency

\n\n

The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a merge operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal “wrapping” contracts and their size is bounded by the number (and height) of source-code contracts in the program.

\n\n

An eidetic contract is an N-ary tree (for N > 0):

\n\n\n\n

For example, the (vectorof point/c) source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the Collapsible Contracts paper has an illustration. Each tree node represents a vectorof contract; these nodes have N=2 children because vectors support reads and writes.

\n\n

A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists — if an implementation comes with a contract-stronger? predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.

\n\n

Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should contract-stronger? try to solve problems that lack polynomial-time solutions?

\n\n

Thankfully, there are at least two “compromise” alternatives.

\n\n

Forgetful space-efficiency

\n\n\n\n\n\n
\n

“Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.” — Space-Efficient Manifest Contracts

\n\n

The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:

\n\n
    \n
  1. the client obligations from the old contract, and
    2. \n
  2. the server obligations from the new contract
\n\n

If not, proceed as usual — by wrapping (an unwrapped value) or raising an error. Every value receives at most one wrapper; this wrapper changes as the value flows to different clients.

\n\n

Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module A exports a function f with contract (-> T1 T2) to module B, and suppose module B shares this function with a few other client modules using different contracts. As f flows to a new client, it keeps the T1 domain check and gets a replacement for the T2 codomain check.

\n\n\n\n

Unfortunately, replacing T2 also means that clients of module B cannot trust the T2 contract. This contract is not checked, and so forgetful contracts miss some errors that would be caught by standard contracts. For the same reason, a bug in module B may go undetected by its clients — even if a later contract reports an issue, the contract system has no memory that B was partly-responsible.

\n\n

Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.

\n\n

Heedful space-efficiency

\n\n

A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set — or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.

\n\n

To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.

\n\n

The heedful method is a compromise between forgetful and eidetic.

\n\n\n\n

For details, see the extended version of Michael’s POPL 2015 paper. Don’t bother searching the conference version — aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.

\n\n

* If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to Michael Ballantyne for explaining this to me.

\n\n

Priorities and Appearances

\n\n

The extended version of Space-Efficient Manifest Contracts introduces the forgetful and heedful methods with extreme modesty. It’s tempting to skip past them and focus on the eidetic method.

\n\n
\n

“Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [Siek and Wadler 2010]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen—warm up exercises for eidetic.” — Space-Efficient Manifest Contracts

\n\n

And yet, at least two other research papers rely on these “strawmen” — or rather, the ideas behind the names.

\n\n

Gradual Typing with Union and Intersection Types, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.

\n\n
\n

“if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one” — Gradual Typing with Union and Intersection Types

\n\n

This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.

\n\n
\n

“removing these casts preserves the soundness of the evaluation while reducing the number of them”

\n

“while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits” — Gradual Typing with Union and Intersection Types

\n\n\n\n\n\n\n

Big Types in Little Runtime, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their transient semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.

\n\n

That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value’s set. When a mismatch occurs, the semantics tries to report every type in the set that relates to the mismatch.

\n\n

And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in ICFP 2018. Where will we see them next?

")) ((? . 33) f post (u . "Turnstile Mailing List") (? . 33) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/11/30/turnstile-mailing-list/index.html" . unix) (u . "/blog/2018/11/30/turnstile-mailing-list/") (u . "2018-11-30T14:55:30") (? . 35) (? . 63) (c (u . "announcement") c (u . "turnstile") c (u . "Author: Stephen Chang")) (u . "\n

Turnstile now has a mailing list: https://groups.google.com/forum/#!forum/turnstile-users

") #t (u . "\n

Turnstile now has a mailing list: https://groups.google.com/forum/#!forum/turnstile-users

\n\n\n

There is also a #turnstile channel on the Racket Slack.

")) ((? . 34) f post (u . "Defining Local Bindings in Turnstile Languages") (? . 34) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/index.html" . unix) (u . "/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/") (u . "2018-10-22T15:05:17") (? . 78) (? . 35) (c (u . "turnstile") c (u . "tutorial") c (u . "language") c (u . "dsl") c (u . "Author: Sam Caldwell")) (u . "\n

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

\n\n

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

\n\n

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

\n\n

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

") #t (u . "\n

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

\n\n

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

\n\n

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

\n\n

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

\n\n\n

The code for this blog post can be found in this gist. To run it, you will need the Turnstile package, which can be installed with raco pkg install\nturnstile.

\n\n

Turnstile: Typechecking Intertwined with Elaboration

\n\n

Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.

\n\n

While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.

\n\n

Type & Elaboration Rules

\n\n

To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.

\n\n

Functions

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
(define-type-constructor → #:arity >= 1)\n(define-typed-syntax (λ ([x:id (~datum :) τ_in:type] ...) e) ≫\n  [[x ≫ x- : τ_in.norm] ... ⊢ e ≫ e- ⇒ τ_out]\n  -------------------------------------------------\n  [⊢ (#%plain-lambda- (x- ...) e-) ⇒ (→ τ_in.norm ... τ_out)])\n
\n
\n
\n\n

Looking at this item by item, we see:

\n\n
    \n
  1. define-type-constructor creates a new type. Here, we say the → requires at least one parameter.
    2. \n
  2. define-typed-syntax is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.
    3. \n
  3. The next part is a syntax-pattern describing the the shape of the syntax this rule applies to. In this case, we’re defining λ as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body e. The type syntax class is provided by Turnstile, and describes the surface syntax of types (such as those created with define-type-constructor); internal operations over types use the expanded version of the type, which is accessed via the norm attribute.
    4. \n
  4. The chevron ≫ on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.
    5. \n
  5. The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.
    6. \n
  6. Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (⊢). For each formal x, this lets us know what type x has when we find a reference to it in e. In this rule, we are saying “while checking the right-hand-side, assume x—which elaborates to x-—has type τ_in, for each triple in the input syntax (signified by the ellipses ...)”. More on the “elaborates to x-” below.
    7. \n
  7. To the right of the turnstile, we write the expression we are checking, e, and patterns e- and τ_out matching the elaboration of e and its type, respectively.
    8. \n
  8. After the dashes comes the conclusion, which begins with ⊢. The next part specifies the elaboration of the term. Here, the meaning of the typed λ is given in terms of Racket’s #%plain-lambda. Turnstile uses the convention of a - suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using postfix-in, such as in (require (postfix-in - racket/base)) to bind #%plain-lambda-.
    9. \n
  9. Finally, we give the type of the term to the right of the ⇒, referring to pattern variables bound in the premises.
\n\n

Renaming Typed Variables

\n\n

Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier x with type τ, Turnstile binds x to a macro that knows τ when it expands. References to x now become references to that macro, and expanding them provides access to τ. Concretely, the underlying Racket code implementing this behavior looks roughly like this:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
(let ([x- (assign-type (generate-temporary #'x) #'τ)])\n  (let-syntax ([x (make-rename-transformer x-)])\n    ... expand and check forms that may reference x ...))\n
\n
\n
\n\n

The type τ is attached as metadata for a new identifier x-, which is what x will transform to at any reference site. In order for this to work, x- must be distinct from x—hence the generate-temporary—to avoid an infinite expansion loop.

\n\n

Application

\n\n

We can define a version of #%app that type checks function applications to accompany our typed λ:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7
\n
\n 
(define-typed-syntax (#%app e_fn e_arg ...) ≫\n  [⊢ e_fn ≫ e_fn- ⇒ (~→ τ_in ... τ_out)]\n  #:fail-unless (stx-length=? #'[τ_in ...] #'[e_arg ...])\n                (num-args-fail-msg #'e_fn #'[τ_in ...] #'[e_arg ...])\n  [⊢ e_arg ≫ e_arg- ⇐ τ_in] ...\n  --------\n  [⊢ (#%plain-app- e_fn- e_arg- ...) ⇒ τ_out])\n
\n
\n
\n\n
    \n
  1. The syntax pattern on the first line describes the shape of applications.
    2. \n
  2. On the second line, we pattern match the result of expanding and checking e_fn, checking that it produces an arrow type. More specifically, when we defined the arrow type → above, define-type-constructor also implicitly defined a pattern expander ~→ (which uses the Racket ~ prefix convention for syntax patterns) that matches instances of the type.
    3. \n
  3. The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.
    4. \n
  4. Line 5 checks that each argument expression has the required type. Turnstile uses bidirectional typechecking rules, which either infer the type of a term or checks that a term satisfies a given type. We write ⇐ τ_in in the premise to switch to checking mode.
    5. \n
  5. Finally, typed function application elaborates to Racket’s function application, #%plain-app, with the usual suffix, and produces type τ_out for the application
\n\n

We can try out these new typed forms on a few examples:

\n\n\n\n

Extending Our Language with Local Bindings

\n\n

When writing functional programs, we often want to name various sub-computations. One way to do that is with a let construct, which Turnstile allows us to easily create:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
(define-typed-syntax (let ([x:id e-x] ...) e-body) ≫\n  [⊢ e-x ≫ e-x- ⇒ τ-x] ...\n  [[x ≫ x- : τ-x] ... ⊢ e-body ≫ e-body- ⇒ τ-body]\n  -------------------------------------\n  [⊢ (let- ([x- e-x-] ...) e-body-) ⇒ τ-body])\n
\n
\n
\n\n

Unsurprisingly, this looks very similar to the definition of λ above. Now we can write functions with named intermediate results:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
(λ ([x : Int])\n  (let ([almost-there (+ x 1)])\n    (+ almost-there 1)))\n
\n
\n
\n\n

However, in Racket it’s common to name such intermediate results using define rather than let. In fact, it’s prescribed by the style guide. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
(λ ([x : Int])\n  (define almost-there (+ x 1))\n  (+ almost-there 1))\n
\n
\n
\n\n

Unfortunately, this is not nearly as easy to do in Turnstile as let.

\n\n

Sequences

\n\n

At first glance, the issue seems to be that the definition of λ above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of λ to allow its body to be a sequence.

\n\n

The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like λ:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
(define-typed-syntax (begin e ...+) ≫\n  [⊢ e ≫ e- ⇒ τ] ...\n  #:with τ-final (stx-last #'(τ ...))\n  -----------------------------------\n  [⊢ (begin- e- ...) ⇒ τ-final])\n
\n
\n
\n\n

This directs type checking to:

\n\n
    \n
  1. Check each e in the sequence individually, obtaining an expanded e- and inferred type τ for each.
    2. \n
  2. Take the last type in the sequence and call it τ-final; Turnstile allows using syntax-parse directives such as #:with as premises.
    3. \n
  3. Expand to Racket’s begin (with the usual - suffix) and give the whole expression the type of the last term in the body.
\n\n

Now, we can use begin in a revised definition of λ. The new rule takes a non-empty sequence of forms in the body and wraps them in our new begin form for typechecking.

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4
\n
\n 
(define-typed-syntax (λ ([x:id (~datum :) τ_in:type] ...) e ...+) ≫\n  [[x ≫ x- : τ_in.norm] ... ⊢ (begin e ...) ≫ e- ⇒ τ_out]\n  -------------------------------------------------\n  [⊢ (#%plain-lambda- (x- ...) e-) ⇒ (→ τ_in.norm ... τ_out)])\n
\n
\n
\n\n

Now we need a way to include definitions in these sequences and we’re set!

\n\n

The Difficulty With Define

\n\n

If we think about how type information is communicated between a binder and its reference we can see why define is a different beast than let

\n\n
(let ([x 5]) (+ x 1))\n       ^        ^\n       |        |- TO HERE\nFROM HERE
\n\n

When the rule for our let is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the ⊢, as in our rules for λ and let above.

\n\n

Define is different.

\n\n
(define x 5)\n        ^\n        |------ TO WHERE?\nFROM HERE
\n\n

The problem is apparent: we can’t see where the reference to x occurs! The information about the binding needs to escape from the define to the surrounding context. In other words, when we implement define, we don’t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the x binding and its type to the surrounding context.

\n\n

Above, in the subsection on “Renaming Typed Variables”, we saw that the context in Turnstile type rules is implemented as syntax transformers with let-like scope (created with let-syntax). One idea would be to mimic this approach, but instead of using let-syntax to achieve let-like scope, use define-syntax to achieve define-like scope.

\n\n

Fortunately for us, someone has already tried their hand at writing a define form for Turnstile languages using a define-syntax rename, found in the Turnstile examples. We can take that as our starting point:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11
\n
\n 
(define-base-type Void)\n(define- a-deep-dark-void (#%app- void-))\n(define-typed-syntax (define x:id e) ≫\n  [⊢ e ≫ e- ⇒ τ]\n  #:with x- (assign-type (generate-temporary #'x) #'τ #:wrap? #f)\n  -----------------------------------------------------\n  [⊢ (begin-\n       (define-syntax x (make-variable-like-transformer #'x-))\n       (define- x- e-)\n       a-deep-dark-void)\n     ⇒ Void])\n
\n
\n
\n\n

Let’s break it down.

\n\n
    \n
  1. Create a new type, Void, to assign definitions.
    2. \n
  2. Create a constant to serve as the canonical value of type Void.
    3. \n
  3. Define a new typed form, define, used as in (define x e).
    4. \n
  4. Check the type of the expression e, getting its expansion e- and type τ.
    5. \n
  5. Create a new name, x-, and attach the type τ as metadata.
    6. \n
  6. Expand to Racket’s begin. Unlike let, begin does not create a new scope; definitions inside a begin are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.
    7. \n
  7. Create a macro binding for x that rewrites to x-. By using a define-like form, the macro has the same scoping rules as define, so it will apply to references to x in the surrounding context—exactly what we want. (We are using make-variable-like-transformer to avoid the special treatment the expander gives to rename-transformers. The specifics are beyond the scope of this post.)
    8. \n
  8. Define x- to refer to the supplied expression. Note that here define- is Racket’s define.
    9. \n
  9. Keep the result of evaluating this form in line with the type by yielding a value of type Void.
\n\n

This implementation of define gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4
\n
\n 
;; module top level\n(define x 5)\n(+ x 1)\n;;=> 6\n
\n
\n
\n\n

Unfortunately, we encounter a problem if we try to create local definitions:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
(define add2\n  (λ ([x : Int])\n     (define almost (+ x 1))\n     (+ almost 1)))\n;;==> almost: unbound identifier...\n
\n
\n
\n\n

Pointing to the reference on the final line. The problem is that our define and begin forms are not interacting in the way we might have hoped.

\n\n

When we expand the body of the function above, we associate x with type Int then start checking the body, wrapped in a begin:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3
\n
\n 
(begin\n  (define almost (+ x 1))\n  (+ almost 1))\n
\n
\n
\n\n

Consulting the definition of begin, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of define, yielding

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4
\n
\n 
(begin-\n  (define-syntax almost ...)\n  (define- almost- ...)\n  a-deep-dark-void)\n
\n
\n
\n\n

Crucially, the expansion of our define form stops at this point, without examining the begin- form and its contained definitions. The interface through which Turnstile invokes the macro expander, local-expand, takes a parameter referred to as the stop list for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.

\n\n

The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named erased, that serves (only) to orchestrate stopping expansion. So, the output of our define form actually looks like

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
(erased\n  (begin-\n    (define-syntax almost ...)\n    (define- almost- ...)\n    a-deep-dark-void))\n
\n
\n
\n\n

And since Turnstile includes erased in the stop list for local-expand, expansion stops before analyzing the rest of the output. The point of all this erased business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.

\n\n

Control returns to the begin transformer, which turns to checking/expanding the subsequent (+ almost 1), where it will encounter the identifier almost without a corresponding binding. Even though our define form produced a binding as part of its output, the expander hasn’t actually analyzed it before reaching the reference in the next expression.

\n\n

The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add define to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn’t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We’re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket’s syntax model.

\n\n

Internal Definition Contexts

\n\n

In order for the (+ almost 1) expression from above to successfully typecheck/expand, we need to be able to associate almost with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile’s interface doesn’t suit this scenario.

\n\n

Racket has the notion of an internal definition context that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.

\n\n

When using local-expand, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then local-expand-ing references such as the above one should work out. Normally, Turnstile calls local-expand internally in accordance with the type rules we write down, but in order to use our own definition context we’re going to have to call it ourselves.

\n\n

We can create a definition context with syntax-local-make-definition-context, as in

\n\n
\n \n \n \n \n
\n
\n 
1
\n
\n 
(define def-ctx (syntax-local-make-definition-context))\n
\n
\n
\n\n

And then (imperatively) add bindings to def-ctx with syntax-local-bind-syntaxes. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier means. Passing #f corresponds to a run-time/phase 0 binding, such as that of a procedure argument, let, or define; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6
\n
\n 
(define-for-syntax (int-def-ctx-bind-type-rename! x x- t ctx)\n  (syntax-local-bind-syntaxes (list x)\n                              #`(make-variable-like-transformer\n                                 (assign-type #'#,x- #'#,t #:wrap? #f))\n                              ctx)\n  (syntax-local-bind-syntaxes (list x-) #f ctx))\n
\n
\n
\n\n

The first call binds x to a transformer that renames to x-; the second lets the expander know that we are taking care of making sure that x- will actually be bound to something.

\n\n

Our define form must communicate the information needed to call int-def-ctx-bind-type-rename! back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of define that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.

\n\n

Concretely, define will expand to define/intermediate, which will in turn expand to what define originally expanded to:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15
\n
\n 
(define-typed-syntax (define x:id e) ≫\n  [⊢ e ≫ e- ⇒ τ]\n  #:with x- (generate-temporary #'x)\n  #:with x+ (syntax-local-identifier-as-binding #'x)\n  --------------------------------------------------\n  [⊢ (define/intermediate x+ x- τ e-) ⇒ Void])\n\n(define-syntax (define/intermediate stx)\n  (syntax-parse stx\n    [(_ x:id x-:id τ e)\n     #:with x-/τ (assign-type #'x- #'τ #:wrap? #f)\n     #'(begin-\n         (define-syntax x (make-variable-like-transformer #'x-/τ))\n         (define- x- e)\n         a-deep-dark-void)]))\n
\n
\n
\n\n

(The reason we create an x+ using syntax-local-identifier-as-binding is due to a bug in the expander. The explanation is rather involved and frankly I only barely understand what’s going on myself (if at all), so let’s just leave it at that and move on.)

\n\n

Then, for each form e in a sequence, we can call local-expand with def-ctx and then check the expansion, e-, for define/intermediate. In those cases, we can use int-def-ctx-bind-type-rename! to add it to the context. The procedure add-bindings-to-ctx! performs this check on an expanded form e- (remembering that Turnstile will wrap the output of define in an erased macro):

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7\n8
\n
\n 
(define-for-syntax (add-bindings-to-ctx! e- def-ctx)\n  (syntax-parse e-\n        #:literals (erased)\n        [(erased (define/intermediate x:id x-:id τ e-))\n         (int-def-ctx-bind-type-rename! #'x #'x- #'τ def-ctx)]\n        [_\n         ;; when e expands to something other than a definition there's nothing to bind\n         (void)]))\n
\n
\n
\n\n

We now have the key ingredients to define a procedure, walk/bind, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form e:

\n\n
    \n
  1. local-expand using our internal definition context, resulting in an e-.
    2. \n
  2. Retrieve the type of e from the metadata of e- using Turnstile’s typeof helper.
    3. \n
  3. Check if e defined a binding, in which case add it to the context.
\n\n

Aggregating the expanded syntax and type of each form as we go along, we get

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14
\n
\n 
(define-for-syntax (walk/bind e...)\n  (define def-ctx (syntax-local-make-definition-context))\n  (define unique (gensym 'walk/bind))\n  (define-values (rev-e-... rev-τ...)\n    (for/fold ([rev-e-... '()]\n               [rev-τ... '()])\n              ([e (in-syntax e...)])\n      (define e- (local-expand e (list unique) (list #'erased) def-ctx))\n      (define τ (typeof e-))\n      (add-bindings-to-ctx! e- def-ctx)\n      (values (cons e- rev-e-...)\n              (cons τ rev-τ...))))\n  (values (reverse rev-e-...)\n          (reverse rev-τ...)))\n
\n
\n
\n\n

The value unique and its use as an argument is dictated by the documentation of local-expand: “For a particular internal-definition context, generate a unique value and put it into a list for context-v.” By using #'erased in the stop list for local-expand, we stop expansion at the same points that Turnstile does.

\n\n

Now we can implement begin in terms of walk/bind:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
(define-typed-syntax (begin e ...+) ≫\n  #:do [(define-values (e-... τ...) (walk/bind #'(e ...)))]\n  #:with τ-final (last τ...)\n  --------------------\n  [⊢ (begin- #,@e-...) ⇒ τ-final])\n
\n
\n
\n\n

and voilà!

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6
\n
\n 
(define (add2 [x : Int])\n  (define almost (+ x 1))\n  (+ almost 1))\n\n(add2 3)\n;;=> 5\n
\n
\n
\n\n

But Wait, There’s More

\n\n

I believe this design is can be dropped in ‘as-is’ and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:

\n\n\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7\n8
\n
\n 
(define-syntax (define/memo stx)\n  (syntax-parse stx\n    [(_ (f [x (~datum :) τ] ...) e ...+)\n     #'(begin\n         (define memo ... memo table ...)\n         (define (f [x : τ] ...)\n           ... check memo table ...\n           e ...))]))\n
\n
\n
\n\n

Finally, there’s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don’t have to muck around with syntax-local-bind-syntaxes and friends. If you have any ideas on this front, feel free to reach out.

\n")) ((? . 36) f post (u . "[Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/)") (? . 36) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/index.html" . unix) (u . "/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/") (u . "2016-11-30T15:24:45") (? . 42) (? . 43) (c (u . "tutorial") c (u . "semantics") c (u . "books") c (u . "Author: Jonathan Schuster")) (? . 5) #f (? . 5)) ((? . 37) f post (u . "[How to prove a compiler fully abstract (cross-post)](https://dbp.io/essays/2018-04-19-how-to-prove-a-compiler-fully-abstract.html)") (? . 37) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/04/23/-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html/index.html" . unix) (u . "/blog/2018/04/23/-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html/") (u . "2018-04-23T10:07:48") (? . 19) (? . 59) (c (u . "Author: Daniel Patterson")) (? . 5) #f (? . 5)) ((? . 38) f post (u . "Scoping in R") (? . 38) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/09/10/scoping-in-r/index.html" . unix) (u . "/blog/2019/09/10/scoping-in-r/") (u . "2019-09-10T10:00:00") (? . 11) (? . 71) (c (u . "scope") c (u . "r") c (u . "Author: Ming-Ho Yee")) (u . "\n

In the previous post of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: is R lexically or dynamically scoped?

") #t (u . "\n

In the previous post of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: is R lexically or dynamically scoped?

\n\n\n

Recall the example program from before:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7
\n
\n 
x <- 1\nf <- function() x\ng <- function() {\n  x <- 2\n  f()\n}\ng() # what does this return?\n
\n
\n
\n\n

Let’s examine what happens when we run this example. First, we create a mapping for x in the top-level environment. On line 2, we define a function f, which returns the value of some x. On line 3, we define a function g, which creates a new mapping for x, and then calls f. Note that the assignment on line 4 does not update the definition on line 1.

\n\n

When f is called, it needs to look up the value of x. In other words, does the reference of x on line 2 refer to the assignment on line 1 or the assignment on line 4? If f returns 1, then the behaviour matches lexical scoping. If it returns 2, then the behaviour matches dynamic scoping.

\n\n

When we run this example, the result is 1. This implies that R is lexically scoped.

\n\n

But there’s more to this story. In the rest of this blog post, I’ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.

\n\n

The next and final part of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.

\n\n

R is lexically scoped, but…

\n\n

In Evaluating the Design of the R Language,1 Morandat, Hill, Osvald, and Vitek write:

\n\n
\n

As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.

\n\n

In other words, R provides many different “escape hatches”—ways to bypass lexical scoping. Additionally, even without escape hatches, some of R’s functionality can be surprising.

\n\n

Functions, environments, and variables in R

\n\n

Before we look at some examples, I think it’s useful to briefly discuss some of the core concepts in R that relate to scoping.

\n\n\n\n

From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.2

\n\n

On the other hand, variable lookup depends on the run-time state of the program—names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.

\n\n

The following examples will make this clear.

\n\n

Examples

\n\n

Adding variable mappings at run time

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7
\n
\n 
x <- 1\nf <- function() {\n  g <- function() x\n  x <- 2\n  g()\n}\nf() # 2\n
\n
\n
\n\n

When f is called, it creates a function g that returns x, assigns 2 to x, and then calls g. When g is called, it looks up x. Since no mapping is found in g’s environment, it searches in the enclosing environment (f’s), and finds that x has value 2. Therefore, g returns 2.

\n\n

Note that the x on line 3 is resolved only when function g is called, not when it is defined. However, when g is defined, its environment has a reference to f’s environment. Therefore, as long as x is defined before g is called, the lookup will always succeed.

\n\n

Here’s a second example:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7\n8
\n
\n 
x <- 1\nf <- function(b) {\n  if (b)\n    x <- 2\n  x\n}\nf(TRUE)  # 2\nf(FALSE) # 1\n
\n
\n
\n\n

f is a function that branches on its argument, b. If b evaluates to true, then the expression x <- 2 is evaluated, and a mapping for x is created in f’s environment. Otherwise, no mapping is created.

\n\n

When we look up the value of x on line 5, R will first search the function’s environment. If b evaluated to true, then R will find a value for x, which is 2. Otherwise, R will search in the enclosing environment of f, and find that x is 1.

\n\n

Both of these examples vaguely resemble dynamic scoping, in that x takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.

\n\n

Function lookup

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5
\n
\n 
f <- function(x) x\ng <- function(f) {\n  f(0) # not an error\n}\ng(42) # 0\n
\n
\n
\n\n

R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.

\n\n

In this example, we call g with the argument 42, which is not a function. Then, in the body of g, we call f(0) on line 3, which requires looking up f. Although there is an f in the environment of g, its value is 42, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so f(0) returns 0.

\n\n

This behaviour exists because c is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.

\n\n

Super assignment

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7\n8
\n
\n 
x <- 0\nf <- function() {\n  x <- 1\n  x <<- 2\n  x\n}\nf() # 1\nx   # 2\n
\n
\n
\n\n

<<- is the “super assignment” operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.

\n\n

In the above program, we define x to be 0 at the top level, and then define the function f. When we call f on line 7, it assigns 1 to x on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, f returns x, which is looked up from the local environment and has value 1. Finally, line 8 looks up x from the top level environment, which has value 2.

\n\n

Evaluating arbitrary code

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10
\n
\n 
x <- 1\nf <- function(t) {\n  eval(parse(text = t))\n  x\n}\ng <- function() {\n  x <- 2\n  f(\"x <- 0\")\n}\ng() # 0\n
\n
\n
\n\n

R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument t, which happens to be the string \"x <- 0\". Then, when line 4 executes, the lookup of x returns 0.

\n\n

Simulating dynamic scope

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7\n8\n9
\n
\n 
x <- 1\nf <- function() {\n  get(\"x\", envir = parent.frame())\n}\ng <- function() {\n  x <- 2\n  f()\n}\ng() # 2\n
\n
\n
\n\n

On line 3, we perform an explicit variable lookup for x, but we do so in the environment parent.frame(), which refers to the calling function’s environment, in this case, g’s environment.. Therefore, the lookup returns 2.

\n\n

Note that R has a similarly named function, parent.env(e) which returns the enclosing environment of the given environment e.

\n\n

Constructing an arbitrary environment

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11
\n
\n 
x <- 1\nf <- function() {\n  e <- new.env()\n  e$x <- 3\n  get(\"x\", envir = e)\n}\ng <- function() {\n  x <- 2\n  f()\n}\ng() # 3\n
\n
\n
\n\n

When f is called, it constructs a new environment, e, which is initially empty. (By default, its enclosing environment is the current environment, which is f’s.) Next, on line 4, it directly adds a mapping to that environment, assigning 3 to x. Then, on line 5, the lookup is explicitly done in environment e, so f returns 3.

\n\n

Deleting mappings

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10
\n
\n 
x <- 1\nf <- function() {\n  rm(\"x\", envir = parent.env(environment()))\n  x\n}\ng <- function() {\n  x <- 2\n  f()\n}\ng() # Error in f() : object 'x' not found\n
\n
\n
\n\n

Not only is it possible to dynamically add and modify mappings in R, but it is also possible to delete mappings. This is what line 3 does: it explicitly removes the mapping for x from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when f is called, the lookup of x fails and an error is raised.

\n\n

Infinite loop during variable lookup

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7
\n
\n 
enva <- new.env()\nenvb <- new.env()\nparent.env(enva) <- envb\nparent.env(envb) <- enva\nf <- function() x\nenvironment(f) <- enva\nf()\n
\n
\n
\n\n

In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.

\n\n

On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: enva’s enclosing environment is envb, and envb’s enclosing environment is enva.

\n\n

On line 5, we define a function with a free variable, x, but on line 6, we set f’s environment to be enva. Finally, we call f.

\n\n

When the body of f is evaluated, it needs to look up x. Lookup starts in f’s environment, which we set to be enva. Since no mapping for x is found, lookup continues in enva’s enclosing environment, which is envb. However, envb is also empty, so lookup continues in its enclosing environment, which is enva, and now lookup results in an infinite loop.

\n\n

An intuition for scoping in R

\n\n

Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18
\n
\n 
# example 1\nx <- 1\nf <- function() {\n  g <- function() x\n  x <- 2\n  g()\n}\nf() # 2\n\n# example 2\nx <- 1\nf <- function(b) {\n  if (b)\n    x <- 2\n  x\n}\nf(TRUE)  # 2\nf(FALSE) # 1\n
\n
\n
\n\n

It seems that x takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What’s missing from our intuition?

\n\n

The key insight is that R is function scoped. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <- does not create a new scope, and it is more useful to think of it as a mutation on the current environment. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)

\n\n

In a sense, it might be more accurate to say that R environments are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.

\n\n

Discussion

\n\n

All of this might make you a little uncomfortable, and uncertain about R’s scoping rules.

\n\n

On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply “lexical-like” behaviour, and the creators have confirmed that lexical scoping was the intent.

\n\n

On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment—but this is not consistent with R’s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.

\n\n

This ambiguity shows up in a paper,3 where the authors write:

\n\n
\n

Furthermore, because variable scoping in R is dynamic and can be modified at the language level […] it cannot be trivially guaranteed that x is going to point to the same data structure throughout the entire execution of the loop.

\n\n

It is true that a variable x may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.

\n\n

It is true that variable lookup is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of dynamic scope, then it would be fair to say that R is dynamically scoped.

\n\n

But if your definition of dynamic scope is “a variable is bound to the most recent assignment during the program’s execution,” then it is not correct to say R is dynamically scoped.

\n\n

I think we have this ambiguity because scope (the places in a program where a variable can be referenced) and variable lookup or name resolution (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.

\n\n

Personally, I prefer the definition of scope that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is both lexically and dynamically scoped, or that R is neither lexically and dynamically scoped.

\n\n

I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language—when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.

\n\n

Regardless, it is very easy to miscommunicate, so I think it’s important to be very clear and make sure you and your audience know what definitions of scoping you’re using!

\n\n

Conclusion

\n\n

This entire adventure started when we were working on a paper,4 and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!

\n\n

This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.

\n\n

To summarize, I would say that scope refers to the places in a program where a variable is visible and can be referenced. Under lexical scoping, the scope of a variable is determined by the lexical (i.e., textual) structure of a program. Under dynamic scoping, a variable is bound to the most recent value assigned to that variable, i.e., the most recent assignment during the program’s execution.

\n\n

I would say that R aims to be lexically scoped—it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.

\n\n

Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!

\n\n

If you want more content about R and scoping, the third and final part of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.

\n\n

Edited 2020/02/21: For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), this blog post has some nice examples and diagrams.

\n\n

I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.

\n\n

If you liked this post, you may also be interested in the following Twitter threads about R: one, two and three.

\n\n
\n\n

References

\n\n
\n
    \n
  1. \n

    F. Morandat, B. Hill, L. Osvald, J. Vitek. “Evaluating the Design of the R Language,” in Proceedings of the European Conference on Object-Oriented Programming (ECOOP), 2012. [DOI][Available online] ↩

    2. \n
  2. \n

    R. Gentleman and R. Ihaka. “Lexical Scope and Statistical Computing”, Journal of Computational and Graphical Statistics, vol. 9, no. 3, 2000. [DOI][Available online] ↩

    3. \n
  3. \n

    L. Stadler, A. Welc, C. Humer, and M. Jordan. “Optimizing R Language Execution via Aggressive Speculation,” in Proceedings of the Symposium on Dynamic Languages (DLS), 2016. [DOI] ↩

    4. \n
  4. \n

    O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. “R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,” in Proceedings of the Symposium on Dynamic Languages (DLS), 2019. To appear. [Available online] ↩

")) ((? . 39) f post (u . "PLISS: Learn About PL Implementation in a Castle") (? . 39) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/index.html" . unix) (u . "/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/") (u . "2019-03-09T14:40:16") (? . 54) (? . 31) (c (u . "event") c (u . "lectures") c (u . "castle") c (u . "language implementation") c (u . "Author: Alexi Turcotte")) (u . "\n

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

\n\n

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

") #t (u . "\n

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

\n\n

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

\n\n\n

\"PLISS

\n\n

The school will feature ten speakers from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:

\n\n\n\n

Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.

\n\n

If I may, I attended the first PLISS in 2017 and can’t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it’s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee’s testimonial here.

\n\n

More information can be found at:

\n\n

https://pliss2019.github.io

\n\n

(No, really, it’s in a castle. Look at the pictures.)

")) ((? . 40) f post (u . "NEPLS on May 31st at UMass, Amherst") (? . 40) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/index.html" . unix) (u . "/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/") (u . "2016-05-03T08:21:07") (? . 67) (? . 69) (c (u . "NEPLS") c (u . "conference") c (u . "Author: Gabriel Scherer")) (u . "\n

It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!

") #t (u . "\n

It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!

\n\n\n
\n

The next New England Programming Languages and Systems Symposium will take place on Tuesday, May 31st 2016 at University of Massachusetts, Amherst. Please mark it in your calendars!

\n

The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.

\n

Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.

\n

The submission deadline is Tuesday, May 17th. Send your proposal to talks@nepls.org.

\n

More details about NEPLS are available on the NEPLS webpage:

\n

http://www.nepls.org/

\n\n

I’m personally fond of such regional events, which is a great time to learn about the research around us in a less formal and exhausting setting than a 200-attendees conference.

\n\n

If you are in the area, please consider applying to talk about your work. If one of your colleague is working on something you find exciting, please invite them to apply!

")) ((? . 41) f post (u . "Understanding Constructive Galois Connections") (? . 41) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/11/16/understanding-constructive-galois-connections/index.html" . unix) (u . "/blog/2016/11/16/understanding-constructive-galois-connections/") (u . "2016-11-16T00:00:00") (? . 13) (? . 42) (c (u . "icfp") c (u . "galois connection") c (u . "adjunction") c (u . "category theory") c (u . "math") c (u . "Author: Max New")) (u . "\n

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

") #t (u . "\n

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

\n\n\n

History

\n\n

I won’t go into much of the details of the paper, because I think it is quite well written, but here’s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of Abstract Interpretation. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the “on-paper” proofs, and offload much of the details to the proof assistant as mere calculation.

\n\n

David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient St. Petersburg, Florida) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, “Kleisli” and “Constructive” Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.

\n\n

Eventually, my usual trawling of Mathoverflow and nlab led me to a not-quite generalization to categories and interestingly a de-generalization to sets that helped me immensely to understand the theorem.

\n\n

Since I know that the original theorem is a bit technical, I’ll explain the de-generalization to sets here, which I hope will help to understand their theorem.

\n\n

Functions and Relations

\n\n

Let’s start with the “Kleisli Arrows”, which are monotone functions \\(f : A \\to P(B) \\) where \\(A,B \\) are posets and \\(P(B)\\) represents the poset of downward-closed subsets of \\(B \\).

\n\n

Now to “de-posetize” this, we’ll take sets \\(X,Y \\) and let \\(P(Y) \\) mean the powerset of \\(Y\\), that is the set of all subsets of \\(Y \\). Then a function \\(f : X \\to P(Y) \\) is actually exactly the same thing as a relation \\(R \\subset X \\times Y \\). From \\(f :\nX \\to P(Y) \\) we can take \\(R = \\{(x,y) \\in X\\times Y | y\\in f(x)\\} \\) and from \\(R\\) we can construct \\(f(x) = \\{y \\in Y | (x,y) \\in R \\}\\).

\n\n

Furthermore, the “Kleisli composition” is the same as composition of relations. If \\(R \\subset X \\times Y \\) and \\(Q \\subset Y \\times Z\n\\), then the composition is defined as \\[ (R;Q) = \\{(x,z) \\in X \\times Z | \\exists y\\in Y. (x,y) \\in R \\land (y,z) \\in Q\\}\\]

\n\n

Then the next thing we need to understand is what is the de-generalization of “Kleisli Galois connection”? Well, Galois connections are an instance of what’s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any “universe” that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I’m talking about is called \\(\\texttt{Rel}\\), and it consists of sets, relations between sets and inclusion of relations, i.e. that one relation is a subset of another.

\n\n

Then what does it mean to have an adjunction between two relations \\(R \\subset X \\times Y, Q \\subset Y \\times X\\)? Taking apart the definition it just means

\n\n

\\begin{align}\\tag{1} \\Delta(X) \\subset R;Q \\end{align} \\begin{align}\\tag{2} Q;R \\subset \\Delta(Y) \\end{align}

\n\n

where \\(\\Delta \\) means the diagonal, or equality relation on the set:

\n\n

\\[\\Delta(X) = \\{(x_1,x_2) \\in X | x_1 = x_2 \\} \\]

\n\n

So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \\(x \\in X\\), there exists a \\(y \\in Y \\) such that \\((x,y) \\in R \\) and \\((y,x) \\in Q\\). This tells us for one that \\(R \\) is a “right-total” relation and \\(Q \\) is a “left-total” relation. Every \\(x \\) is related to some \\( y\\) by \\( R \\) and \\( Q\\).

\n\n

If we unwind (2), we get that for any \\(y,y' \\in Y\\) if there’s some \\(x \\in X \\) such that \\((x,y) \\in R \\) and \\((y',x) \\in Q \\) then actually \\(y = y')\\). This one is a bit more mysterious, but first, let’s see what this tells us about the relationship between \\(R\\) and \\(Q \\).

\n\n

If \\((x,y) \\in R \\), then by (1) there’s some \\(y' \\in Y\\) so that \\((x,y') \\in R \\) and \\((y',x) \\in Q\\). Then, by (2) we know that \\(y = y'\\), so we’ve shown that if \\((x,y) \\in R \\) then \\((y,x)\n\\in Q\\). Then a completely symmetric argument shows that if \\((y,x)\n\\in Q \\) then \\((x,y)\\in R\\)! So we’ve discovered that actually \\(Q \\) is just the opposite relation of \\(R \\).

\n\n

Then if we look at (2) again but replace the \\(Q\\)’s by flipped \\(R\\)’s we get that for any \\(y,y' \\in Y\\), if there’s some \\(x\n\\in X\\) such that \\((x,y) \\in R \\) and \\((x,y')\\in R\\) then \\(y\n= y'\\), which tells us that \\(R \\) is a partial function, i.e., that every \\(x \\) is related to at most one \\(y \\) by \\(R \\).

\n\n

You may recognize it now, our \\(R \\subset X \\times Y \\) is just a function, and saying \\(R, Q\\) are adjoint is exactly the same as saying that \\(Q = R^{\\text{op}}\\) and \\(R \\) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!

\n\n

Constructive Galois Connections

\n\n

Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have “monotone relations”, i.e., downward-closed \\(R \\subset A^{\\text{op}} \\times B \\). Then you can interpret the definition of adjunction in that universe and get that it’s the same as a Kleisli Galois connection and that a similar argument to the above shows that the “left adjoint” is represented by a monotone function \\(f : A \\to B \\):

\n\n

\\[R = \\{(x,y) | y \\le f(x) \\} \\]

\n\n

Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.

\n\n

In fact, we get a little extra from what’s mentioned in their paper, which is that the “right adjoint” is represented by \\(f \\) as well but in the opposite way:

\n\n

\\[Q = \\{(y,x) | f(x) \\le y \\}\\]

\n\n

Category Theory Post Scriptum

\n\n

If you’re interested in Category theory, here’s a more technical addendum.

\n\n

Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?

\n\n

Well, first we need a generalization from relations to downward-closed relations to what are called distributors or profunctors. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?

\n\n

The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \\(R : C\n\\not\\to D \\), which is called (for mind-boggling reasons) Cauchy completeness. Viewing sets and posets as special categories, it turns out that they always have this property, and that’s why the theorem worked out for those adjunctions.

")) ((? . 30) f post (u . "Datalog for Static Analysis") (? . 30) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/02/21/datalog-for-static-analysis/index.html" . unix) (u . "/blog/2017/02/21/datalog-for-static-analysis/") (u . "2017-02-21T12:58:27") (? . 29) (? . 25) (c (u . "HOPL") c (u . "Author: Ben Greenman")) (? . 5) #t (u . "\n\n

Datalog is an old DSL that frequently appears in work on static analysis. This edition of HOPL 2017 explores the origins of Datalog in general, its early use in program analysis, and why Datalog remains a useful tool.

\n\n

Full notes:

\n\n\n\n
\n\n

Datalog as a language was introduced by 1978 (its semantic foundations date back to 1976). It is predicate logic as a database query language. The traditional view of a Datalog program is a time invariant transformation over the time varying data stored in an external database.

\n\n

In the early 1990’s, Uwe Aβmann designed a graph rewriting systems (EARS) that could:

\n\n
    \n
  1. Uniformly express various problems in static analysis
    2. \n
  2. Systematically derive efficient solutions to such problems.
\n\n

(Prior work had derived the same solutions with ad-hoc methods.) Aβmann’s system is equivalent to Datalog.

\n\n

In 1993, Reps used the \n CORAL deductive database (an implementation of Datalog) to derive an on-demand (read: lazy) implementation of program slicing from a specification of the slicing problem.

\n\n

Both Aβmann’s and Reps work appeared in 1994. This was the first time Datalog had been used to implement a static analysis.

\n\n

Researchers continue to use Datalog because:

\n\n\n\n

For an excellent description of how Datalog can benefit static analysis, see the introduction to Rep’s paper.

")) ((? . 43) f post (u . "Measuring the submission/review balance") (? . 43) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/12/17/measuring-the-submission-review-balance/index.html" . unix) (u . "/blog/2016/12/17/measuring-the-submission-review-balance/") (u . "2016-12-17T16:33:10") (? . 36) (? . 61) (c (u . "Author: Gabriel Scherer")) (u . "\n

How do researchers know whether they are doing “enough” or “too many” reviews? A measurable goal is to be review-neutral: to have demanded, through our submissions, as many reviews as we have produced as reviewers.

") #t (u . "\n

How do researchers know whether they are doing “enough” or “too many” reviews? A measurable goal is to be review-neutral: to have demanded, through our submissions, as many reviews as we have produced as reviewers.

\n\n\n

Reviewing is good

\n\n

I like to review academic papers. It is a very rewarding activity in many different ways. One gets to serve the academic community, helping it function smoothly. One gets a chance at acquiring a much better understanding of someone else’s work than idle paper-skimming allows. One gets to send feedback to our colleagues and help them improve their work and its presentation — it is also an essential way in which we can participate to the formation of student researchers all over the world. Finally, doing reviews helped me develop the skill the judge someone else’s work and of forcing oneself to come up with a decisive opinion — it is surprisingly difficult and only comes with training.

\n\n

Doing reviews is also fairly time-consuming. I noticed that the time I spend on each review is generally stable (excursions into previous or related work excluded): around one day and a half for conference reviews, and at least twice more for journal reviews — I’m sure other people have wildly different figures, but I would expect it to be a noticeable time commitment in any case. (Workshop reviews are much easier, at least for the formats I have seen of 2-page extended abstracts, I’d say one hour per review.)

\n\n

How many reviews?

\n\n

Because it is so time-consuming, deciding whether to say “yes” or “no” to invitations to review a new paper is not easy: in general I want to say “yes” (unless I can tell that I will not enjoy reading the paper at all), but it is not reasonable to say “yes” all the time, because I also need to spend time on other things. When should I say “no” because I have done “too many” reviews already?

\n\n

We can count the number of reviews that we have done, and we can also estimate the number of reviews that we have demanded of others through our submissions. A natural goal for researchers is to produce at least as many reviews as they demand; if everyone reached this goal, the peer-review system would be at equilibrium without imposing too much of a workload on anyone.

\n\n

To estimate the number of reviews a researcher demanded from their peers, you can sum, for each of their submissions to a peer-reviewed venue, the number of reviews that they received, divided by the total number of authors of the submissions.

\n\n

\\[ \\sum_{p \\in \\mathtt{Submissions}} \\frac{\\mathtt{reviews}(p)}{\\mathtt{authors}(p)} \\]

\n\n

Out of curiosity, I just measured this balance for myself: over my years doing research I have “demanded” 10 workshop reviews and 28.5 conference reviews, and “produced” 6 workshop reviews and 17 conference reviews. If you think that an article would interest me, you shouldn’t feel bad about asking me to review it, for now. (On the other hand, my balance this year is positive, so I wouldn’t feel to bad about refusing if I had to.)

\n\n

Of course, a researcher’s balance is highly dependent on where they are in their academic career — maybe more so that on their personal choices. Students are supposed to submit articles, but are offered few opportunities for doing reviews. When they are invited to do reviews, it is often as sub-reviewer, one review at a time. More established researchers participate in program committees, where they have to do a fair amount of reviews at once — ten to twenty can be typical in Programming Languages conferences. This means that one naturally starts with a deficit of reviews, and that the opportunity to become balanced or positive only comes over the years.

\n\n

(There is much more that could be said about the dynamics of the submission/review balance. I think the idea that a single person should be neutral should not be taken too seriously, because the dynamics are so complex. For example, some people stop doing reviews with a negative balance (students going to the industry for example), so long-time researchers necessarily have a very positive balance that may make short-time researchers balance considerations mostly irrelevant. Another thing is that there is no point doing more reviews than required by the submission flow, and that doing more reviews would build up more reviewing debt under this neutrality criterion — you can never have everyone positive.)

\n\n

Quality

\n\n

This is only a comment on the quantitative aspects of reviewing. Much more important is the qualitative part: are the reviews you receive and produce good reviews? (There is no objective definition of what a good review is; I like reviews that are constructive, help improve the work and its presentation, and catch mistakes.) For a given paper, one or a few very good reviews is more helpful than many bad reviews, so one should not compromise on the quality of one’s reviews in order to reach a quantitative goal.

\n\n

Advice for students?

\n\n

While proof-reading this post (thanks!), Ben asked some questions that may be of interest to others — mostly students, I suppose.

\n\n
\n

If I want to be review-neutral, but I have to accumulate a “review debt” before I can start reviewing, does this mean I should accept my first opportunity to review and every one that follows (until I’m neutral)?

\n\n

The answer is of course “no”: one should never feel forced to accept reviews. On the other hand, I do think that it is worthwhile for PhD students to take advantage of the reviews they are offered, so “saying yes most of the time” sounds like a reasonable strategy to me — this is just a personal opinion. Some reasons:

\n\n\n\n

On the other hand, I think you should not accept reviews at times when you cannot invest enough work in the review, or when doing so would be detrimental to you — whether you are on a deadline, or under too much pressure, or have conflicting commitments, etc. This is more important than anything about a submission/review balance.

\n\n
\n

Do you have any ideas for how young researchers / new researchers can reduce their “review footprint”? For example, is it possible to volunteer for reviews?

\n\n

Yes, you can volunteer for reviews by telling the colleagues in your lab that you would be interested in doing reviews and that they should consider giving you some.

\n\n

(With the increased use of double-blind submission processes, it is becoming more difficult to pass conference reviews to external researchers. This means that students are relatively unlikely to receive review offers from outside their close colleagues.)

\n\n

Besides doing more reviews, the two other factors one could in theory play with are: submitting less papers, and having more co-authors. I think there is something to be said for the first one: one reason to not submit unfinished, buggy or topically-inappropriate articles is that it has a review cost. The second factor should not be considered, I think: “did this person contribute to the work?” should weight infinitely more for co-authorship decisions.

\n\n

Note: Another thing you can ask for is reading reviews other people received. I think that reading reviews is also very helpful for research beginners — whether reviews of one’s own work or someone else’s. In particular, I wouldn’t know how to write reviews if I hadn’t had the opportunity to read reviews before that. If someone you are close to receives reviews, you should consider asking them whether you could have a look.

\n\n
\n

Is being a student volunteer at a conference equal to “one review”?

\n\n

I think it is a distinct form of academic service. I don’t know how to measure the “conference organization cost” we impose to our academic colleagues. (If there are around 500 attendants to a typical Programming Languages conference, it means that for every 500 conferences you attend you should organize one all by yourself.)

")) ((? . 44) f post (u . "Transient for Optional and Keyword Functions") (? . 44) 1731622765 (p+ #"/home/runner/work/website/website/blog/2020/11/12/transient-for-optional-and-keyword-functions/index.html" . unix) (u . "/blog/2020/11/12/transient-for-optional-and-keyword-functions/") (u . "2020-11-12T10:15:16") (? . 72) (? . 45) (c (u . "typed racket") c (u . "transient") c (u . "Author: Ben Greenman")) (u . "\n

A short adventure into the depths of optional and/or keyword functions in Racket.

") #t (u . "\n

A short adventure into the depths of optional and/or keyword functions in Racket.

\n\n\n
\n\n

Transient, or rather the Transient semantics for a mixed-typed language, is one way to let statically-typed code safely interact with untyped code. You can read all about it in Michael Vitousek’s 2019 dissertation or my 2020 dissertation, and you can see how it compares to other mixed-typed semantics here. The idea is to give up on behavioral type guarantees and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called shape checks; for example:

\n\n\n\n

Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type T into a shape check, and we have a function with type T that needs to check its inputs. The question is how to actually do the check in Racket v7.9.

\n\n

In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if T = (-> Symbol Boolean) then we need to check the shape symbol? of the domain type Symbol:

\n\n
;; source code\n(: f (-> Symbol Boolean))\n(define (f sym)\n  (eq? sym 'hola))\n\n;; ===>\n\n;; imaginary (but realistic) rewritten code\n(define (f sym)\n  (assert sym symbol?)\n  (eq? sym 'hola))
\n\n

A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type T accepts 1 to 3 inputs:

\n\n
;; source code\n(: g (->* [#:a Boolean] [Symbol #:c Void] Symbol))\n(define (g #:a a [b 'b] #:c [c #f])\n  (if a b (if c 'left 'right)))\n\n;; ===>\n\n;; imaginary, unrealistic rewritten code\n(define (g #:a a [b 'b] #:c [c #f])\n  (assert a boolean?)\n  (assert b symbol?)\n  (assert c void?)\n  (if a b (if c 'left 'right)))
\n\n

Good — we basically know what we want. If the Racket core language had optional and keyword functions, then we’d be done.

\n\n

But no, Racket expands these optional/keyword functions into primitive lambda and case-lambda forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.

\n\n

Let’s keep digging.

\n\n

From now on, “Shallow” or “Shallow TR” refers to my implementation of Transient for Typed Racket (TR). We’ll talk about Shallow instead of “Transient” in case future work reveals a better way to implement the Transient idea.

\n\n

False Start: Follow the Type

\n\n

Beware — Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function g above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional #:c keyword lead to a shape-check failure because the variable c gets the default value #f instead of a void value. Concretely, the third assert from above fails:

\n\n
(define (g #:a a [b 'b] #:c [c #f])\n  ....\n  (assert c void?) ;; fails if c is the #f default value\n  ....)
\n\n

The problem arises from subtyping. According to the annotations, the function g has an external type that is less precise than the internal type that validates the function body:

\n\n
;; external type T\n(: g (->* [#:a Boolean] [Symbol #:c Void] Symbol))\n\n;; internal type T2, subtype of external (T2 <: T), validates body\n(: g (->* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))
\n\n

Thanks to this external / internal distinction, the following easy rewrite idea, Solution 0, fails. Despite the failure, this first solution is a useful starting point for a success.

\n\n

Solution 0, Step 1: Mimic the Typechecker

\n\n

Shallow TR uses the same type checker as classic Deep TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.

\n\n

Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.

\n\n

Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (typecheck/tc-expr-unit.rkt). The omitted code (….) does actual type checking:

\n\n
(define (tc-expr/check/internal form expected-type)\n  ....\n  (syntax-parse form\n    #:literal-sets (kernel-literals tc-expr-literals)\n    ....\n    [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^)\n    ....]\n    [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^)\n    ....]
\n\n

Ok! Those two patterns say a lot about the expansion of optional and keyword functions:

\n\n
    \n
  1. Both forms expand to a let-values that binds one function fun.
    2. \n
  2. TR uses the syntax classes kw-lambda^ and opt-lambda^ to tell these particular let-values apart from others.
\n\n

Shallow TR can use exactly these patterns to recognize optional/keyword functions.

\n\n

Solution 0, Step 2: Parse the Domain Type

\n\n

Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function’s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:

\n\n
    [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^)\n    ....]
\n\n

First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The type-of function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, fun, has a type. Great.

\n\n

Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:

\n\n\n\n

Solution 0, Step 3: Insert a Shape Check

\n\n

Once we have the target fun expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse fun to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting fun goes something like this:

\n\n
(syntax-parse fun\n  [(#%plain-lambda formals . body)\n   #:with (shape-check ...)\n          (make-shape-checks #'formals (type-of fun))\n   #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])
\n\n

The rewritten function executes shape checks immediately, and then proceeds with the body after validating each actual parameter.

\n\n

On the Trail: optkey Expansion

\n\n

Our Solution 0 fails because the type of the fun expression that it gets from the type-checked code is an external type. In terms of the g function from above, Solution 0 uses the type Void instead of the internal type (U Void #f) to check the c parameter. To get internal types, we need to look closer at fun and the rest of the optional/keyword expansion.

\n\n

Let’s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.

\n\n

If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function X create a module test.rkt like this:

\n\n
#lang racket/base\n\n(define _ignore\n  (let ()\n    X\n    (void)))
\n\n

Invoke the expander with raco expand test.rkt > test.rkt.txt and explore the generated .txt file.

\n\n

Example 1: mandatory keyword

\n\n

The source is a function with one mandatory positional argument and one optional positional argument.

\n\n
(lambda (x [y 0])\n  (+ x y))
\n\n

Expansion generates a case-lambda that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function F that expects two arguments, resolves defaults in a different way, and executes the function body.

\n\n
(let-values (((F)\n              (lambda (x2 y1)\n                (let-values (((x) x2))\n                  (let-values (((y) (if '#f '0 y1)))\n                    (let-values () (#%app + x y)))))))\n  (case-lambda\n   ((x) (#%app F x '0))\n   ((x y1) (#%app F x y1))))
\n\n

Note: the expression (if ’#f ’0 y1) in the generated F function is equal to y1 alone. In general, the if is for default expressions. (Unlike Python, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a #f test. A general-purpose optimizer can remove this test before the code runs.

\n\n

Example 2:

\n\n

The source is a function with one mandatory positional argument and one mandatory keyword argument:

\n\n
(lambda (x #:y y)\n  (+ x y))
\n\n

Expansion generates several functions:

\n\n\n\n

The important piece for Shallow TR is the F0 function because the goal of rewriting is to protect the original function body against untyped inputs.

\n\n
(let-values (((F0)\n              (lambda (y1 x3)\n                (let-values (((x) x3))\n                  (let-values (((y) y1))\n                    (let-values () (#%app + x y)))))))\n  (let-values (((F1)\n                (lambda (given-kws given-args x3)\n                  (let-values (((y1) (#%app car given-args)))\n                    (#%app F0 y1 x3)))))\n    (#%app\n     lifted/2\n     (lambda (given-kws given-argc)\n       (if (#%app = given-argc '3)\n         (let-values (((l2571) given-kws))\n           (if (#%app pair? l2571)\n             (if (#%app eq? (#%app car l2571) '#:y)\n               (#%app null? (#%app cdr l2571))\n               '#f)\n             '#f))\n         '#f))\n     (case-lambda\n      ((given-kws given-args x)\n       (#%app F1 given-kws given-args x)))\n     '(#:y)\n     '(#:y))))
\n\n

Example 3:

\n\n

The source is a function with one mandatory positional argument and one optional keyword argument:

\n\n
(lambda (x #:y [y 0])\n  (+ x y))
\n\n

Expansion again generates several functions:

\n\n\n\n

Again, the F0 function is the focal point for Shallow TR rewriting.

\n\n
(let-values (((F0)\n              (lambda (y1 x3)\n                (let-values (((x) x3))\n                  (let-values (((y) (if '#f '0 y1)))\n                    (let-values () (#%app + x y)))))))\n  (let-values (((F1)\n                (lambda (given-kws given-args x3)\n                  (let-values (((y2) (#%app pair? given-kws)))\n                    (let-values (((y1)\n                                  (if y2 (#%app car given-args) '0)))\n                      (#%app F0 y1 x3))))))\n    (#%app\n     make-optional-keyword-procedure\n     (lambda (given-kws given-argc)\n       (if (#%app = given-argc '3)\n         (let-values (((l1571) given-kws))\n           (let-values (((l1571)\n                         (if (#%app null? l1571)\n                           l1571\n                           (if (#%app eq? (#%app car l1571) '#:y)\n                             (#%app cdr l1571)\n                             l1571))))\n             (#%app null? l1571)))\n         '#f))\n     (case-lambda\n      ((given-kws given-args x)\n       (#%app F1 given-kws given-args x)))\n     null\n     '(#:y)\n     (case-lambda\n      ((x) (#%app F1 null null x))))))
\n\n

Solution: The Shallow TR Rewrite Strategy

\n\n

All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a let-values form:

\n\n
(let-values (((f) fun)) . body)
\n\n

Furthermore, the generated fun is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the fun from Example 3 again; it has formal parameters for the keyword arg. and the mandatory arg., and one let-values to resolve each parameter:

\n\n
  (lambda (y1 x3)\n    (let-values (((x) x3))\n      (let-values (((y) (if '#f '0 y1)))\n        (let-values () (#%app + x y)))))
\n\n

Another experiment with type-of shows that the right-hand side of each let-values has an internal type annotation. Excellent! Both (type-of x3) and (type-of (if ’#f ’0 y1)) are the right types for shape checks. Shallow TR can:

\n\n\n\n

This should work! In fact, we can do slightly better:

\n\n\n\n

Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.

\n\n

A Problem with Methods and a Bugfix

\n\n

Currently, Shallow TR rewrites optional and keyword functions using the let-values plan described above. Each formal parameter has one let-values binding, and the type on each bound expression defines the shape check.

\n\n

Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we fixed the class expander.

\n\n

First, here is a class with one method that runs correctly. The method f accepts an optional positional argument x; the default value of x is the current value of the field my-num (fields are mutable):

\n\n
(define c0%\n  (class object%\n    (super-new)\n    (field (my-num 2))\n    (define/public (f [x my-num])\n      (+ x x))))
\n\n

Second, here is a similar method that fails. This time, the default is an immediate value 2:

\n\n
(define c1%\n  (class object%\n    (super-new)\n    (define/public (f [x 2])\n      (+ x x))))
\n\n

Running a call (send o1 f) used to raise a shape-check failure about a strange value:

\n\n
\n

shape error: Expected a real number, got #<unsafe-undefined>

\n\n

What is going on?

\n\n

It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:

\n\n
(lambda (x [y z])\n  (+ x y))
\n\n

Expansion generates a function F0 that checks for the undefined value, and an outer case-lambda that supplies undefined when the default is needed:

\n\n
(let-values (((F0)\n              (lambda (x2 y1)\n                (let-values (((x) x2))\n                  (let-values (((y)\n                                (if (#%app eq? y1 unsafe-undefined)\n                                  z\n                                  y1)))\n                    (let-values () (#%app + x y)))))))\n  (case-lambda\n   ((x) (#%app F0 x unsafe-undefined))\n   ((x y1) (#%app F0 x y1))))
\n\n

That’s the normal way that unsafe-undefined shows up: the expander for optional/keyword functions looks for default expressions vs. default values and uses the undefined value for expressions.

\n\n

Three other facts conspired to make the problem with optional methods:

\n\n
    \n
  1. Typed Racket also looks for default expressions vs. default values (search for immediate-default here). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the unsafe-undefined value (search for -Unsafe-Undefined here (kw) and here (opt)).
    2. \n
  2. The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.
    3. \n
  3. Shallow TR pushes default expression checks (if test default-expr supplied-arg) to the supplied-arg instead of wrapping the whole if form.
\n\n

In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound—but harmless because the false assumption was guarded by an if test for unsafe-undefined. Running Shallow TR revealed the unsoundness with its eager shape check.

\n\n

Again, the resolution was to fix the class expander (racket/racket #3182). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.

\n\n

Lessons

\n\n
    \n
  1. Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.
    2. \n
  2. Digging into the expansion is sometimes necessary. There are at least three places that do so—the class expander, TR, and Shallow TR—and unfortunately they all need to cooperate.
    3. \n
  3. The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of my dissertation lists them all.
")) ((? . 46) f post (u . "A few cores too many") (? . 46) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/08/03/a-few-cores-too-many/index.html" . unix) (u . "/blog/2016/08/03/a-few-cores-too-many/") (u . "2016-08-03T14:09:02") (? . 52) (? . 47) (c (u . "performance") c (u . "benchmarking") c (u . "lost time") c (u . "Author: Ben Greenman")) (u . "\n

Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.

") #t (u . "\n

Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.

\n\n\n

Last year, we proposed a method for evaluating the performance of gradual type systems based on measuring every possible configuration of typed and untyped code that a programmer might explore (pdf). Given the freedom that gradual typing offers, this is the only realistic way to measure the performance of a gradual type system.

\n\n

But it is a lot to measure! While developing the method, we spent over 3 months benchmarking a total of 75,844 configurations. Each configuration is a complete program and some gradual typings caused some programs to slow by 50x or even 100x, so many of these configurations took minutes to run.

\n\n

The next question we asked was naturally “how can we scale this method to large software projects?” In our case, the number of gradually typed configurations scaled exponentially with the number of modules. Current gradual type system for Python and JavaScript are exponential in the number of variables in the program.

\n\n

We explored two solutions:

\n\n
    \n
  1. Max New began work on a prediction model (inspired by work on software product lines) to estimate the performance of 2^N configurations after polynomially-many measurements.
    2. \n
  2. Asumu Takikawa and I shopped for a multi-core computer.
\n\n

By Thanksgiving, we had bought a Linux machine with 2 AMD Opteron 6376 2.3GHz processors (16 cores each) and put it to work running benchmarks on 29 cores simultaneously. Life was good.

\n\n

Later that winter, Max implemented a prediction algorithm. The basic idea was to focus on boundaries between modules and isolate their effect on performance. If two modules are untyped, their boundary will have zero cost. If the same two modules are typed, their boundary might result in an overall performance improvement due to type-driven optimizations. And if one module is typed and the other untyped, their boundary will suffer some cost of type checking at runtime. In general a program with N modules has at most N(N - 1) / 2 internal boundaries, so it is far more time-efficient to measure only the boundaries than to benchmark 2^N gradually typed configurations.

\n\n

Fast-forward to March, we had a prototype prediction algorithm and it was time to test. Again using 29 cores (because, why not), we gathered cost/benefit numbers for one 4-module benchmark and used them to predict performance for its 16 configurations. The results were not very good.

\n\n
\"Figure\n

Figure 1: True running time vs. predicted running time for 16 configurations

\n\n

Those green circles are the ground truth, the average running time after 5 iterations of each config. The blue triangles are what we predicted. Except for configurations 0 and 8, the triangles are FAR off from the truth. Many are even negative … obviously the algorithm needs work.

\n\n

But then, out of frustration, desperation, or just good luck, Max compared the predictions to ground truth data gathered on a single core, leaving the other 31 cores idle.

\n\n
\"Figure\n

Figure 2: Predictions made using measurements from a single core

\n\n

First off, the red “sequential truth” dots are slightly closer to the predicted triangles. Second — and this is the scary part — the red dots are very different from the green dots. Running on 1 core vs. 29 cores should not change the measurements!

\n\n

From here we tried increasing the running time of the benchmark, removing I/O and system calls, checking for hyperthreading (ARM cores don’t support it), and even changing the cores’ CPU governor. The hope was that results taken from 1 core could match results from N cores, for some N > 1. It turns out N = 2 was stable, but even for N = 3 we found graphs like the following:

\n\n
\"Figure\n

Figure 3: exact running times. Same-colored dots in each column should be tightly clustered.

\n\n

This data is for the same 16 configurations as the previous two graphs. Green dots are exact running times measured with 25 cores. Red dots are running times measured with 1 core. The red dots are much closer together, and always unimodal. The green dots are evidence that maybe the 32-core machine has, as Jan Vitek put it, 30 cores too many.

\n\n
\n

“Oh my. You think it’ll never happen to you. Well, now I’ve learned my lesson.”

\n\n

And so, we said goodbye to the last 4 months of data and started over running at most two cores. The new results are all stable, but still we keep pinching ourselves.

\n\n

P.S. the results from POPL 2016 are just fine, as they were not taken on the new machine running more than 2 cores. If you have time to confirm, that data is in our artifact and in the gradual-typing-performance repo.

")) ((? . 45) f post (u . "Deep and Shallow Types") (? . 45) 1731622765 (p+ #"/home/runner/work/website/website/blog/2020/12/23/deep-and-shallow-types/index.html" . unix) (u . "/blog/2020/12/23/deep-and-shallow-types/") (u . "2020-12-23T18:21:55") (? . 44) (? . 0) (c (u . "dissertation") c (u . "migratory typing") c (u . "Author: Ben Greenman")) (u . "\n

I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:

\n\n\n\n

To the PRL: thanks for a wonderful 6.5 years.

") #t (u . "\n

I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:

\n\n\n\n

To the PRL: thanks for a wonderful 6.5 years.

\n\n\n

Abstract

\n\n
\n

The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound’’ mixed-typed languages from others that enforce type obligations locally rather than globally.

\n

In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.

\n

My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.

\n

I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.

\n\n

Next year, I’ll be a CI Fellow at Brown.

")) ((? . 48) f post (u . "Report: PLISS 2017") (? . 48) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/06/05/report-pliss-2017/index.html" . unix) (u . "/blog/2017/06/05/report-pliss-2017/") (u . "2017-06-05T15:47:59") (? . 81) (? . 49) (c (u . "pliss") c (u . "event") c (u . "Author: Ming-Ho Yee")) (u . "\n

Two weeks ago, I attended the first Programming Language Implementation Summer School, held in beautiful Bertinoro, Italy.

\n\n

The goal of PLISS was “to prepare early graduate students and advanced undergraduates for research in the field,” and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.

") #t (u . "\n

Two weeks ago, I attended the first Programming Language Implementation Summer School, held in beautiful Bertinoro, Italy.

\n\n

The goal of PLISS was “to prepare early graduate students and advanced undergraduates for research in the field,” and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.

\n\n\n

The Talks

\n\n

With the amount of technical content at PLISS, there was easily something for everyone. Jan Vitek and Laurence Tratt gave lectures that included hands-on exercises where we worked on JITs. Suresh Jagannathan dived into the operational semantics of a distributed system, so we could reason about different weak consistency models. Francesco Logozzo gave us a whirlwind tour of abstract interpretation.

\n\n

Most of my favorite talks included some form of extra content, such as exercises, live-coding presentations, or demos. I found it really helpful to write actual code and apply what I had just learned, or to look at some concrete examples. The examples and exercises also helped with the pacing, as actively listening to four 90-minute talks every day is exhausting!

\n\n

Off the top of my head, these were some of my favorite talks:

\n\n\n\n

If you’re disappointed that you couldn’t see these talks, don’t worry! The talks were recorded and will be posted very shortly.

\n\n

The People

\n\n

But there’s more to PLISS than the talks. I’m referring to networking, or the opportunity to get out and talk to other people about research.

\n\n

As an early graduate student, I’ve been given a lot of advice about talking to people at conferences and the importance of the “hallway track.” I still have difficulty doing this at an actual conference, like PLDI or ECOOP. When there are hundreds of attendees, or when people already know each other and are in conversation groups, I find it difficult to approach them.

\n\n

This was not the case at PLISS. There were fewer attendees: about fifty students and a dozen speakers. There was a good mix of undergraduate, master’s, first-year PhD, and more senior PhD students. All our breakfasts, lunches, and breaks were together, so we would see the same people again and again, and inevitably start to learn each other’s names. The speakers would also be among us, and there was a good ratio of speakers to students for discussions and mealtime mentoring.

\n\n

I had many opportunities to practice my “research pitch.” I talked to senior students and got advice. I talked to junior students and gave advice. Two different people I talked to about my research pointed me to the same paper to read. I found another student who was working with IFDS, an algorithm I have spent much time trying to understand. And, one day at lunch, my table discovered that we were all working on static analysis!

\n\n

As much as I enjoyed the talks, I think the best part of PLISS was meeting and talking to other people. You can replace talks with videos (but you lose the speaker-audience interaction), and you can replace conversations with other forms of communication. But there isn’t really anything that can replace the serendipity of bumping into someone with a shared interest.

\n\n

The Location

\n\n

Actually, the other best part of PLISS was the location. Italy is a beautiful country with delicious food. And Bertinoro is a small town on the top of a hill, with a breathtaking view of the surrounding area. The lectures were held in a castle at the top of the hill (photo credit: Steve Blackburn). The speakers lived in the castle for the week, while the students lived in the former monastery (seems fitting), which has been renovated into a university residence.

\n\n

Here are my two favorite pictures I took (click for full size):

\n\n

\"View

\n\n

\"Panorama\"

\n\n

Steve Blackburn has more pictures posted on the PLISS website.

\n\n

Final Thoughts

\n\n

PLISS was a wonderful event. Many thanks need to be given to the speakers, organizers, and sponsors, for making this possible!

\n\n

If and when there is a second PLISS, I highly encourage students to apply! You will learn a lot from the lectures, from talking to the speakers, and meeting other students. And if it’s in Bertinoro again, you can enjoy the weather and nice view!

")) ((? . 50) f post (u . "Programming Language Conference in Russia") (? . 50) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/05/24/programming-language-conference-in-russia/index.html" . unix) (u . "/blog/2017/05/24/programming-language-conference-in-russia/") (u . "2017-05-24T12:25:17") (? . 51) (? . 80) (c (u . "Author: Artem Pelenitsyn")) (u . "\n

In April 3—5 I took part into a Russian conference exclusively devoted to programming languages: Programming Languages and Compilers (Google.Translated version of the site). I was a member of organizing committee and had a paper there.

\n\n

This is the first conference in Russia highly focused on our area of PL. At least for the last several decades (I believe, there were conferences of the kind back in USSR). The conference was devoted to the memory of prominent Soviet PL-researcher from Rostov-on-Don, Adolf Fuksman who worked on ideas quite similar to what we know as the aspect-oriented programming back in the 70-s.

") #t (u . "\n

In April 3—5 I took part into a Russian conference exclusively devoted to programming languages: Programming Languages and Compilers (Google.Translated version of the site). I was a member of organizing committee and had a paper there.

\n\n

This is the first conference in Russia highly focused on our area of PL. At least for the last several decades (I believe, there were conferences of the kind back in USSR). The conference was devoted to the memory of prominent Soviet PL-researcher from Rostov-on-Don, Adolf Fuksman who worked on ideas quite similar to what we know as the aspect-oriented programming back in the 70-s.

\n\n\n

We designed and implemented the conference with my colleagues from I.I.Vorovich institute of Mathematics, Mechanics and Computer Science, Southern Federal University (Rostov-on-Don, Russia). We aimed at gathering as much PL-researchers and enthusiasts from academia in Russia as we could. One of the consequences of the aim was a decision to run the conference in Russian. Though we missed expertise from our non-Russian speaking colleagues, we got thorough participation from all over Russia:

\n\n
\n

Saint-Petersburg, Moscow, Novosibirsk, Krasnoyarsk, Ekaterinburg, Kazan, etc.

\n\n

I only mention here the cities with more than 1 mil. population in decreasing in number of conference participants order (and excluding Rostov itself, of course).

\n\n

I particularly liked talks by invited speakers. When searching for ones, we targeted Russians who work at prominent universities and/or have some visibility at the international level. We ended up with two researchers: Ilya Sergey (University College of London) and Ekaterina Komendantskaya (Heriot-Watt U., Edinburg, UK). Interestingly, their talks were quite close to each other:

\n\n\n\n

Both of them talked about types as a logic tool to ensure program correctness.

\n\n

Biggest opening in this conference for me was a team from Languages Tools Laboratory of JetBrains. Surely, you heard about JB: either about their famous IDE, IntelliJ IDEA, or the Kotlin programming language (which, by the way, is endorsed for Android development these days). You may also have noticed that JB become sponsors of ECOOP and OPLSS this year. So we had a whole team of researchers from Saint-Petersburg office of JB. Among their topics: OCaml embedding of miniKanren (some results was presented on ML Workshop 2016), parser combinator libraries for OCaml and constrained graph querying (this is not specifically a PL problem, see arXiv:1502.02242 for details).

\n\n

Otherwise the spectrum of topics presented on the conference was quite broad, here are some:

\n\n\n\n

Full program with links to slides in Russian is available here.

\n\n

Let me mention my submission: that was a joint work with a student of mine on exploring design space for parser combinator libraries using programming language with direct support of effect system, namely Frank. Julia Belyakova also participated in the conference with her work on Coq-certified interpreter for an extension of lambda-calculus with concept-parameters (module-like kind of thing). The follow-up of that work is accepted for FTfJP workshop this year. You can also listen to her on the topic at the NEPLS next week.

\n\n

I hope that we will find sources, time, and, most important, high quality submissions for PLC–2018.

")) ((? . 52) f post (u . "Tutorial: Zero to Sixty in Racket") (? . 52) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/08/02/tutorial-zero-to-sixty-in-racket/index.html" . unix) (u . "/blog/2016/08/02/tutorial-zero-to-sixty-in-racket/") (u . "2016-08-02T01:29:11") (? . 53) (? . 46) (c (u . "Racket") c (u . "tutorial") c (u . "Author: Ben Greenman")) (u . "\n

Racket is excellent for incrementally growing scripts into full-fledged programs.\nThis post steps through the evolution of one small program and highlights the\n Racket tools that enable incremental advances.

") #t (u . "\n

Racket is excellent for incrementally growing scripts into full-fledged programs.\nThis post steps through the evolution of one small program and highlights the\n Racket tools that enable incremental advances.

\n\n\n

\n\n
Why should anyone use Racket?\nThere are two reasons:\n
\n\n
\n
    \n
  1. \n

    You have a problem that can only be solved with Racket’s language-building tools

    2. \n
  2. \n

    Racket is a nice language to program in.\n(Has lexical scope, parentheses, active users...)

\n\n

My favorite part of Racket is how it supports a certain development style of\n evolving scripts into programs.\nWhen I start coding (after design, before debugging), I can focus the problem at hand.\nNext come examples, unit tests, and types to be sure the solution is correct.\nFinally, I worry about aesthetics, efficiency, and how the solution can be used as a library in a larger context.

\n\n

Bottom line: with Racket, my coding is aligned with my priorities.\nAnd as I transition from \"no code\" to \"working code\" to \"robust code\" to \"re-usable code\",\n the program is almost always runnable.

\n\n

Problem: A KWIC Index Production System

\n\n

A KWIC index system\nreads input from a file,\ndivides each line of the file into whitespace-separated words,\nand outputs (in alphabetical order) all circular shifts of all lines.

\n\n

The first circular shift of a line \"A B C\" is the line \"B C A\".\nThe second circular shift is \"C A B\".

\n\n

Building a KWIC index is a historical problem.\nAccording to D.L. Parnas (1972):

\n\n
\n

Except under extreme circumstances (huge data base, no supporting software)\nsuch a system could be implemented by a good programmer within a week or two.

\n\n

See also: Yannis’s Law.

\n\n

Today, I bet only Agda and Scratch\n programmers would need the full two weeks.\nWe’ll be done in 20 minutes.

\n\n

A Script

\n\n

To start, open a file and type:

\n\n
\n \n \n \n
#lang racket
\n\n

You can name the file anything, like kwic.rkt or rkt.kwic or foo.\nRacket doesn’t care,\n but it does need the #lang line to read the contents of the file.

\n\n

Though, you should use the .rkt extension.

\n\n

The first part of the solution is a function to read input from a file into a\n list of strings for further processing.\nThe built-in function file->lines\n does exactly this, but we’ll for-loop instead.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n
(define (kwic-read filename)
  (with-input-from-file filename
    (λ ()
      (for/list ([line (in-lines)])
        line))))
\n\n

When called with a filename like \"heart-of-darkness.txt\", the function\n uses for/list to build a list of lines by reading from a port\n with in-lines.\nThe port is the data from filename, thanks to with-input-from-file.

\n\n

Next is a function to convert a list of strings into a list of lists of words.\nHere we’ll just use library functions.

\n\n
\n \n \n \n \n \n
(define (kwic-split lines)
  (map string-split lines))
\n\n

By default, string-split divides a string into a list of whitespace-separated substrings.\nYou can always supply a different delimiter, or use regexp-split\n to divide by a regular expression.

\n\n

Two tasks left!\nFirst we generate all circular shifts for a list of strings words\n by folding up a list with one shift of words for each word.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(define (all-circular-shifts words)
  (for/fold ([all-shifts (list words)])
            ([i (in-range 1 (length words))])
    (cons (circular-shift (first all-shifts))
          all-shifts)))
\n\n

Second, we alphabetize and print the shifts.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(define (alphabetize all-shifts)
  (sort all-shifts shift<?))
 
(define (shift<? shift1 shift2)
  (match* (shift1 shift2) ; destruct multiple values
   [('() _) ; first list empty, don't care about second
    #t]
   [(_ '()) ; first list non-empty, second empty
    #f]
   [((cons s1 shift1-rest) (cons s2 shift2-rest))
    (or (string<? s1 s2)
        (and (string=? s1 s2)
             (shift<? shift1-rest shift2-rest)))]))
 
(define (kwic-display all-sorted-shifts)
  (define (display-words words)
    (display (first words))
    (for ([word (in-list (cdr words))])
      (display \" \")
      (display word))
    (newline))
  ; for-each is like map, but returns (void)
  (for-each display-words all-sorted-shifts))
\n\n

Gluing it all together, here’s the full script (with type annotations in comments).

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
#lang racket
 
; type Words = (Listof String)
; type Lines = (Listof Words)
 
; Path-String -> (Listof String)
(define (kwic-read filename)
  (with-input-from-file filename
    (λ ()
      (for/list ([line (in-lines)])
        line))))
 
; (Listof String) -> Lines
(define (kwic-split lines)
  (map string-split lines))
 
; Words -> (Listof Words)
(define (all-circular-shifts words)
  (for/fold ([all-shifts (list words)])
            ([i (in-range 1 (length words))])
    (cons (circular-shift (first all-shifts))
          all-shifts)))
 
; Move first element to last position
; Words -> Words
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
; Lines -> Lines
(define (alphabetize all-shifts)
  (sort all-shifts shift<?))
 
; Lexicographic order on equal-length lists of words
; Words Words -> Boolean
(define (shift<? shift1 shift2)
  (match* (shift1 shift2)
   [('() _) ; first list empty, don't care about second
    #t]
   [(_ '()) ; first list non-empty, second empty
    #f]
   [((cons s1 shift1-rest) (cons s2 shift2-rest))
    (or (string<? s1 s2)
        (and (string=? s1 s2)
             (not (null? shift1-rest))
             (shift<? shift1-rest shift2-rest)))]))
 
; Lines -> Void
(define (kwic-display all-sorted-shifts)
  (define (display-words words)
    (display (first words))
    (for ([word (in-list (cdr words))])
      (display \" \")
      (display word))
    (newline))
  (for-each display-words all-sorted-shifts))
 
; Lines -> (Listof Lines)
(define (all-circular-shifts* lines)
  (map all-circular-shifts lines))
 
; Path-String -> Void
(define (kwic-index file-name)
  (define all-lines (kwic-split (kwic-read file-name)))
  (define all-shifts (append* (all-circular-shifts* all-lines)))
  (kwic-display (alphabetize all-shifts)))
 
; End-to-end test
; -> Void
(define (run-test)
  (define test-file \"test.txt\")
  ; Make a file and test
  (unless (file-exists? test-file)
    (with-output-to-file test-file
      (λ ()
        (displayln \"imagine if this\")
        (displayln \"took 2 weeks to write\"))))
  (kwic-index test-file))
 
(run-test)
\n\n

Running the file should print:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
2 weeks to write took
if this imagine
imagine if this
this imagine if
to write took 2 weeks
took 2 weeks to write
weeks to write took 2
write took 2 weeks to
\n\n

Testing and Submodules

\n\n

Any top-level expressions in a file can work as unit tests.\nThe equal? statement below checks whether the first circular shift\n of '(\"A\" \"B\" \"C\") is '(\"B\" \"C\" \"A\").

\n\n
\n \n \n \n \n \n \n \n \n \n
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(equal? (circular-shift '(\"A\" \"B\" \"C\")) '(\"B\" \"C\" \"A\"))
\n\n

Running the file now prints #t to the console, meaning the test passed.\nWe can use error or raise-user-error to make failures easier\n to notice.\nOr we can use the RackUnit testing library.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(require rackunit) ; import the testing library
(check-equal?
  (circular-shift '(\"A\" \"B\" \"C\"))
  '(\"B\" \"C\" \"A\"))
\n\n

These tests run each time the module does.\nIf you prefer to run tests only in a specific context, and not when the\n module is run or imported as a library, you can move them to a separate\n file or into a submodule.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(module+ test ; Open a submodule named 'test'
  (require rackunit)
  (check-equal?
    (circular-shift '(\"A\" \"B\" \"C\"))
    '(\"B\" \"C\" \"A\")))
\n\n

Running the module normally via racket kwic.rkt will not run code\n in the submodule.\nInstead, use raco test to run the tests.

\n\n
\n \n \n \n \n \n \n \n
>  raco test kwic.rkt
raco test: (submod \"kwic.rkt\" test)
1 test passed
\n\n

The reason we used module+, instead of Racket’s module and module*\n forms is that module+ inherits the language and namespace of its\n containing module and can be incrementally extended.\nThis way, we can keep tests near the relevant code.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(module+ test
  (require rackunit))
 
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(module+ test
  (check-equal?
    (circular-shift '(\"A\" \"B\" \"C\"))
    '(\"B\" \"C\" \"A\")))
 
(define (alphabetize all-shifts)
  (sort all-shifts shift<?))
 
(module+ test
  (check-equal?
    (alphabetize '((\"racket\" \"is\")
                   (\"as\")
                   (\"racket\" \"does\")))
    '((\"as\")
      (\"racket\" \"does\")
      (\"racket\" \"is\"))))
\n\n

RackUnit in a\n separate file or test submodule is the unofficial standard for testing\n Racket programs.

\n\n

Recognizing Patterns, Avoiding Repetition

\n\n

Every unit test we’ve written uses check-equal?.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(module+ test
  (check-equal?
    (kwic-split '())
    '())
  (check-equal?
    (kwic-split '(\"hello    world\"))
    '((\"hello\" \"world\")))
  (check-equal?
    (kwic-split '(\" lost \" \" in \" \"space\"))
    '((\"lost\") (\"in\") (\"space\")))
  (check-equal?
    (kwic-split '(\"something\"))
    '()))
\n\n

These tests follow a simple pattern that we can express as a syntax rule.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(module+ test
  (define-syntax-rule (check-equal?* [i o] ...)
    (begin
      (check-equal? i o)
      ...))
 
  (check-equal?*
    [(kwic-split '())
     '()]
    [(kwic-split '(\"hello    world\"))
     '((\"hello\" \"world\"))]
    [(kwic-split '(\" out \" \" in \" \"the ozone\"))
     '((\"out\") (\"in\") (\"the\" \"ozone\"))]
    [(kwic-split '(\"something\"))
     '()]))
\n\n

The ... are not pseudocode!\nThey denote Kleene-star repetition, like a sextile (\"*\") in a regular expression.\nIn this case, the input pattern is a sequence of lists with two S-expressions, i and\n o.\nUses of i and o in the rule must be followed by one ... to splice\n the captured S-expressions into the result.

\n\n

Many languages offer higher-order functions and polymorphism to abstract common\n behaviors.\nSyntax extensions are a different way to avoid repeating yourself.\nAfter 30 years, we are still discovering what syntax extensions are useful for.

\n\n

See this recent Racket mailing list post for some applications.

\n\n

Adding Static Types

\n\n

Changing the #lang line to typed/racket adds static type-checking to our program.\nIf we only change the language and run the code as-is, there will be type errors.\nBut we can use submodules again to incrementally check our design with types.

\n\n

Note: typed regions\n are another way to embed typed code into untyped contexts.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
#lang racket
 
 (module t typed/racket
   ; Need to annotate:
   ; - function parameters
   ; - for-loop return types
 
   (: kwic-read : Path-String -> (Listof String))
   (define (kwic-read filename)
     (with-input-from-file filename
       (λ ()
         (for/list ([line (in-lines)])
                   : (Listof String)
           line))))
 
   ; Next migration step: move other untyped functions here
 
   (provide (all-defined-out)))
 (require 't)
 
 (define (kwic-split lines)
   (map string-split lines))
 
 ; <rest of file omitted>
\n\n

After scooping all functions into the Typed Racket bubble, we can remove the\n submodule declaration and change #lang racket to #lang typed/racket.

\n\n

Finally, a Library

\n\n

Other modules can import our functions if we use a provide statement.\nBy convention, exports belong at the top of a file.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n
 
#lang typed/racket
 
(provide kwic-index)
 
; <definitions here>
\n\n

Then any typed or untyped module can use kwic-index by writing\n (require \"kwic.rkt\").

\n\n

As a finishing touch, we can use the racket/cmdline library\n inside a main submodule to give a basic front-end interface.\nSimilar to module+ test, a module+ main declares code that\n inherits the file’s bindings and language but is only run when the program\n is executaed.

\n\n

Here is the complete typed and tested code listing.\nThe main submodule is at the bottom.

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
#lang typed/racket
(module+ test
  (require typed/rackunit)
 
  (define-syntax-rule (check-equal?* [i o] ...)
    (begin
      (check-equal? i o)
      ...)))
 
(define-type Words (Listof String))
(define-type Lines (Listof Words))
 
(: kwic-read : Path-String -> (Listof String))
(define (kwic-read filename)
  (with-input-from-file filename
    (λ ()
      (for/list ([line (in-lines)])
                : (Listof String)
        line))))
 
(module+ test
  (let ([tmpfile (make-temporary-file)])
    (with-output-to-file tmpfile #:exists 'replace
      (λ ()
        (displayln \"The Nellie,\")
        (displayln \"a cruising yawl,\")
        (displayln \"swung to her anchor without a flutter of sails,\")
        (displayln \"and was at rest.\")))
    (define actual (kwic-read tmpfile))
    (define expect (file->lines tmpfile))
    (delete-file tmpfile)
    (check-equal? actual expect)))
 
(: kwic-split : (Listof String) -> Lines)
(define (kwic-split lines)
  (map #{string-split :: (String -> Words)} lines))
 
(module+ test
  (check-equal?*
    [(kwic-split '())
     '()]
    [(kwic-split '(\"hello    world\"))
     '((\"hello\" \"world\"))]))
 
; Move first element to last position
(: circular-shift : Words -> Words)
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(module+ test
  (check-equal?*
    [(circular-shift '(\"A\" \"B\" \"C\"))
     '(\"B\" \"C\" \"A\")]))
 
(: all-circular-shifts : Words -> (Listof Words))
(define (all-circular-shifts words)
  (for/fold ([all-shifts (list words)])
            ([i (in-range 1 (length words))])
            : (Listof Words)
    (cons (circular-shift (first all-shifts))
          all-shifts)))
 
(module+ test
  (check-equal?*
    [(all-circular-shifts '(\"A\" \"B\" \"C\"))
     '((\"C\" \"A\" \"B\") (\"B\" \"C\" \"A\") (\"A\" \"B\" \"C\"))]))
 
(: alphabetize : Lines -> Lines)
(define (alphabetize all-shifts)
  (sort all-shifts shift<?))
 
(module+ test
  (check-equal?*
    [(alphabetize '((\"A\" \"B\" \"C\") (\"B\" \"C\") (\"A\")))
     '((\"A\") (\"A\" \"B\" \"C\") (\"B\" \"C\"))]))
 
; Lexicographic order on equal-length lists of words
(: shift<? : Words Words -> Boolean)
(define (shift<? shift1 shift2)
  (match* (shift1 shift2)
   [('() _) ; first list empty, don't care about second
    #t]
   [(_ '()) ; first list non-empty, second empty
    #f]
   [((cons s1 shift1-rest) (cons s2 shift2-rest))
    (or (string<? s1 s2)
        (and (string=? s1 s2)
             (shift<? shift1-rest shift2-rest)))]))
 
(module+ test
  (check-equal?*
    [(shift<? '() '())
     #t]
    [(shift<? '(\"A\" \"B\") '(\"A\" \"C\"))
     #t]))
 
(: kwic-display : Lines -> Void)
(define (kwic-display all-sorted-shifts)
  (: display-words : Words -> Void)
  (define (display-words words)
    (display (first words))
    (for ([word (in-list (cdr words))])
      (display \" \")
      (display word))
    (newline))
  (for-each display-words all-sorted-shifts))
 
(module+ test
  (parameterize ([current-output-port (open-output-string)])
    (kwic-display '((\"A\") (\"B\" \"C\")))
    (check-equal?
      (get-output-string (current-output-port))
      \"A\\nB C\\n\")))
 
(: all-circular-shifts* : Lines -> (Listof Lines))
(define (all-circular-shifts* lines)
  (map all-circular-shifts lines))
 
(module+ test
  (check-equal?
    (all-circular-shifts* '((\"A\" \"B\" \"C\") (\"D\")))
    '(((\"C\" \"A\" \"B\") (\"B\" \"C\" \"A\") (\"A\" \"B\" \"C\")) ((\"D\")))))
 
(: kwic-index : Path-String -> Void)
(define (kwic-index file-name)
  (define all-lines (kwic-split (kwic-read file-name)))
  (define all-shifts (append* (all-circular-shifts* all-lines)))
  (kwic-display (alphabetize all-shifts)))
 
(module+ test
  (parameterize ([current-output-port (open-output-string)])
    (define tmpfile (make-temporary-file))
    (with-output-to-file tmpfile #:exists 'replace
      (λ ()
        (displayln \"imagine if this\")
        (displayln \"took 2 weeks to write\")))
    (kwic-index tmpfile)
    (delete-file tmpfile)
    (check-equal?
      (get-output-string (current-output-port))
      (string-join '(
        \"2 weeks to write took\"
        \"if this imagine\"
        \"imagine if this\"
        \"this imagine if\"
        \"to write took 2 weeks\"
        \"took 2 weeks to write\"
        \"weeks to write took 2\"
        \"write took 2 weeks to\\n\") \"\\n\"))))
 
(module+ main
  (require racket/cmdline)
  (: *output-to* (Parameterof Any))
  (define *output-to* (make-parameter #f))
  (command-line
    #:program \"kwic index\"
    #:once-each
    [(\"-o\" \"--output\")
     output-to ; user-supplied input
     \"Write output to file\"
     (*output-to* output-to)] ; update the parameter *output-to*
    #:args (file-name)
    (define output-to (*output-to*)) ; read from parameter *output-to*
    (define out-port
      (if (string? output-to)
        (open-output-file output-to #:exists 'replace)
        (current-output-port)))
    (parameterize ([current-output-port out-port])
      (kwic-index (cast file-name Path-String)))
    (when (string? output-to)
      (close-output-port out-port))))
\n\n

\n\n
Sample interactions:\n
\n\n
\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> racket kwic.rkt
kwic index: expects 1 <file-name> on the command line, given 0 arguments
 
> echo \"It is a truth universally acknowledged\" > pride-and-prejudice.txt
> racket kwic.rkt -o index.out pride-and-prejudice.txt
> wc -l index.out
6 index.out
\n\n

Closing

\n\n

We started with functions, wrote (and quarantined) unit tests,\n reinforced our design with types, and added a command-line interface.\nGoing forward we could add Scribble documentation and share our work as a package.

\n\n

\n\n
For more on building languages with Racket:\n
\n\n
\n
")) ((? . 54) f post (u . "Writing a paper with Scribble") (? . 54) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/02/17/writing-a-paper-with-scribble/index.html" . unix) (u . "/blog/2019/02/17/writing-a-paper-with-scribble/") (u . "2019-02-17T16:20:50") (? . 76) (? . 39) (c (u . "Scribble") c (u . "tutorial") c (u . "Author: Ben Greenman")) (u . "\n

This post explains how to get started using Scribble to write a research paper.

") #t (u . "\n

This post explains how to get started using Scribble to write a research paper.

\n\n\n
\n\n
\n

This post was written using Racket 7.1 and Scribble 1.29

\n\n

Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:

\n\n\n\n

Scribble, the Racket documentation tool, comes with a to-LaTeX compiler and a scribble/acmart library tailored to the new ACM paper format. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short “getting started” guide and links to some larger examples.

\n\n
\n

For a Scribble tutorial, see the links in: Building a Website with Scribble

\n\n

Getting started with scribble/acmart

\n\n

The first line of a scribble/acmart document sets the formatting options (similar to a LaTeX file using acmart.cls). For example, the GPCE 2018 call for papers asks for anonymized sigplan-format submissions with line numbers and 10 point font. The proper Scribble incantation is:

\n\n
#lang scribble/acmart @sigplan @anonymous @review @10pt
\n\n

Next, you may want to import some definitions. If we have a file references.rkt (see below for a definition), we can import it as follows:

\n\n
@require{references.rkt}
\n\n

The third main ingredient is the title and author information:

\n\n
@(define neu (affiliation #:institution \"Northeastern University\"))\n@(define anon (email \"anon@anon.net\"))\n\n@title{Writing a paper with Scribble}\n@author[#:affiliation neu #:email anon]{Ben Greenman}\n\n@; optional: set the author names in the page headers\n@elem[#:style \"Sshortauthors\"]{B. Greenman}
\n\n

The paper is now ready to be written. You can forge ahead with a new section and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and import content from other modules:

\n\n
@include-abstract{abstract.scrbl}\n@include-section{introduction.scrbl}
\n\n

Finally, the main page is a good place to generate the bibliography. Assuming this document imports a file like the references.rkt below, this expression inserts a bibliography titled “References”:

\n\n
@generate-bibliography[#:sec-title \"References\"]
\n\n

To build the document, invoke scribble on the command-line with the --pdf or --latex options:

\n\n
$ raco scribble --pdf FILE.scrbl
\n\n

If all goes well, this command generates a FILE.pdf with properly-linked cross references.

\n\n

Auxiliary Files

\n\n

If you save the code above to a file example.scrbl and save the files below in the same directory, then you should be able to build an example.pdf.

\n\n

These files are available in a slightly different format at this link:

\n\n\n\n

references.rkt

\n\n
#lang racket/base\n\n(provide\n  ~cite citet generate-bibliography\n  fbf-icfp-2009)\n\n(require\n  scriblib/autobib)\n\n(define-cite ~cite citet generate-bibliography\n  #:style author+date-square-bracket-style)\n\n(define icfp \"ICFP\")\n\n(define fbf-icfp-2009\n  (make-bib\n    #:title \"Scribble: Closing the Book on Ad Hoc Documentation Tools\"\n    #:author (authors \"Matthew Flatt\" \"Eli Barzilay\" \"Robert Bruce Findler\")\n    #:location (proceedings-location icfp #:pages '(109 120))\n    #:date 2017))
\n\n

abstract.scrbl

\n\n
#lang scribble/acmart\n\nA simple Scribble document.
\n\n

introduction.scrbl

\n\n
#lang scribble/acmart\n@require{references.rkt}\n\n@; start with `title` instead of `section`, because importing via\n@;  `include-section` shifts all title/section/subsections down one level\n@title{Introduction}\n\nScribble creates a connection between a stand-alone document and the artifact\nit describes@~cite[fbf-icfp-2009].
\n\n

Q. How to debug Scribble error messages?

\n\n

If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.

\n\n

As a compile-time example, adding @ foo to a document produces the message unexpected whitespace after @ and you can either delete the whitespace or change the @ to @\"@\" for a literal @-sign.

\n\n

As a run-time example, adding @(+ 2 2) produces this message:

\n\n
not valid in document body (need a pre-part for decode) in: 4
\n\n

One fix is to convert 4 to a string, as in @~a[(+ 2 2)].

\n\n

But if something goes wrong when Scribble renders a generated document to PDF, the default error output is not likely to help. For example, adding @elem[#:style \"oops\"] to a document produces a giant message:

\n\n
$ raco scribble --pdf FILE.scrbl\n[[ ... 84K of output ... ]]\nOutput written on example.pdf (1 page, 277876 bytes).\nPDF statistics:\n 53 PDF objects out of 1000 (max. 8388607)\n 37 compressed objects within 1 object stream\n 7 named destinations out of 1000 (max. 500000)\n 36877 words of extra memory for PDF output out of 42996 (max. 10000000)\n\nrun-pdflatex: got error exit code\n  context...:\n  [[ ... 17 more lines ... ]]
\n\n

The best way to debug these messages is to ignore them and use a LaTeX compiler directly. For the “oops” mistake, LaTeX stops at the undefined control sequence — giving a hint about how to find the problem:

\n\n
$ raco scribble --latex FILE.scrbl\n$ pdflatex FILE.tex\n[[ ... 12KB of output ... ]]\n! Undefined control sequence.\nl.549 \\oops\n           {}\n?
\n\n

Q. How to add a LaTeX style file?

\n\n

To add extra LaTeX code to the final document, create a new file and include it with the ++style command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).

\n\n
$ raco scribble ++style style.tex --pdf FILE.scrbl
\n\n

Here is an example style file.

\n\n

style.tex

\n\n
\\settopmatter{printfolios=true,printccs=true,printacmref=true}\n% add page numbers etc.\n\n\\overfullrule=1mm\n% draw a black rectangle near lines that overflow the margin
\n\n

Another way to add extra LaTeX code is to add a tex-addition style property to the main title. This second approach makes it easy to include more than one file:

\n\n
#lang scribble/acmart\n\n@require[\n  (only-in scribble/core make-style)\n  (only-in scribble/latex-properties make-tex-addition)]\n\n@(define extra-style-files\n   (list (make-tex-addition \"style.tex\")))\n\n@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble}\n\n@; ....
\n\n

Q. How to make a figure?

\n\n

Use the scriblib/figure library to add figures to a document.

\n\n
@require[pict scriblib/figure]\n@figure[\n  \"fig:fish\"  @; figure tag, see `figure-ref`\n  @elem{A Standard Fish}  @; figure caption, appears below the content\n  @elem{fish = @(standard-fish 90 40)}]  @; content
\n\n

The content of a figure can be almost anything that would work in the toplevel of the document.

\n\n

Q. How to include extra files (pictures, LaTeX)?

\n\n

The ++extra command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.

\n\n

For example, if a document includes the content of an external LaTeX file:

\n\n
@elem[#:style \"input\"]{inline-this.tex}
\n\n

then make sure to build the document with a command like this:

\n\n
$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl
\n\n

inline-this.tex

\n\n
% Raw LaTeX allowed here\n$\\lambda x.\\, x$
\n\n

Q. What about in-line LaTeX?

\n\n

An element with the 'exact-chars style property renders directly to LaTeX.

\n\n
@(define (exact . stuff)\n   @; the style name \"relax\" puts a `\\relax` no-op in front of the stuff\n   (make-element (make-style \"relax\" '(exact-chars)) stuff))\n\n@exact|{$\\lambda x.\\, x$}|\n@; ==> \\relax{$\\lambda x.\\, x$}\n\n@(define ($ . math-stuff)\n   (apply exact (list \"$\" math-stuff \"$\")))\n\n@${\\lambda x.\\, x}\n@; ==> \\relax{$\\lambda x.\\, x$}
\n\n

Creating a #lang for a paper

\n\n

For a Scribble document that is split across multiple files, it can be helpful to make a #lang that provides a common environment. Instead of starting each file with a require, e.g.:

\n\n

paper.scrbl

\n\n
#lang scribble/acmart\n@require[\"references.rkt\" \"helper-functions.rkt\" scriblib/figure]\n\n....
\n\n

files can start with a name that describes their common purpose:

\n\n

paper.scrbl

\n\n
#lang conference-2018-submission\n\n....
\n\n

As a bonus, if the language is defined as a package then the Scribble document can use Racket’s dependency management tools:

\n\n
# to install the paper and interactively install dependencies:\n$ cd conference-2018-submission;\n$ raco pkg install\n\n# To check that the paper builds with no dependency issues:\n$ raco setup --check-pkg-deps conference-2018-submission\n\n# To run all unit tests\n$ raco test -c conference-2018-submission
\n\n

To create a package and language:

\n\n
    \n
  1. Move the Scribble document to a directory with the language name, i.e., conference-2018-submission/
    2. \n
  2. Write a simple info.rkt to configure the package
    3. \n
  3. Create a normal Racket module that exports the common environment
    4. \n
  4. Create a conference-2018-submission/lang/reader.rkt module
\n\n

Details below. For a full example, visit:

\n\n\n\n
\n\n

conference-2018-submission/info.rkt

\n\n

This file defines the basic metadata for a package. For more about info.rkt, see: Tutorial: Creating a Package.

\n\n
#lang info\n(define collection \"conference-2018-submission\")\n(define deps '(\"base\" \"scribble-lib\" \"at-exp-lib\"))\n(define build-deps '(\"racket-doc\" \"scribble-doc\"))\n(define pkg-desc \"Paper for Conference 2018\")\n(define version \"0.1\")
\n\n
\n\n

conference-2018-submission/main.rkt

\n\n

This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the scribble/acmart language, the file “references.rkt”, and the scriblib/figure library.

\n\n
#lang racket/base\n\n(provide\n  (all-from-out\n    scribble/acmart\n    scribble/acmart/lang\n    scriblib/figure\n    \"references.rkt\"))\n\n(require\n  scribble/acmart\n  scribble/acmart/lang\n  scriblib/figure\n  \"references.rkt\")
\n\n
\n\n

conference-2018-submission/lang/reader.rkt

\n\n

This file: (1) tells Racket to use the Scribble reader on #lang conference-2018-submission modules, and (2) wraps the result of such modules in a shape that Scribble expects.

\n\n
#lang s-exp scribble/base/reader\nconference-2018-submission\n#:wrapper1 (lambda (t) (cons 'doc (t)))
\n\n\n\n

These documents use the #lang approach to writing a paper with Scribble. Check their main.rkt for example formatting functions and unit tests, and check the .scrbl files to see how the ideas above look in a larger document.

\n\n\n\n

Finally, this repository provides a tool to start a new Scribble document:

\n\n\n\n

Further Reading

\n\n")) ((? . 55) f post (u . "History of Actors") (? . 55) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/10/19/history-of-actors/index.html" . unix) (u . "/blog/2016/10/19/history-of-actors/") (u . "2016-10-19T17:26:16") (? . 56) (? . 57) (c (u . "history") c (u . "Author: Tony Garnock-Jones")) (u . "\n

Christos Dimoulas is currently teaching a “History of Programming Languages” class at Harvard. The class is, as Christos writes, “definitely not about this”; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.

\n\n

On Monday, I presented “the History of Actors” for the class. I’ve made the written-out talk notes and an annotated bibliography available here.

") #f (u . "\n

Christos Dimoulas is currently teaching a “History of Programming Languages” class at Harvard. The class is, as Christos writes, “definitely not about this”; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.

\n\n

On Monday, I presented “the History of Actors” for the class. I’ve made the written-out talk notes and an annotated bibliography available here.

")) ((? . 58) f post (u . "Gradual Typing Across the Spectrum, part II") (? . 58) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/index.html" . unix) (u . "/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/") (u . "2017-08-22T15:54:06") (? . 65) (? . 75) (c (u . "gradual typing") c (u . "PI meeting") c (u . "Author: Ben Greenman")) (u . "\n

Last week, Northeastern hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:

\n\n

http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html

\n\n

A common thread among the talks was the question: how to convert a research idea into a tool for software developers?

") #t (u . "\n

Last week, Northeastern hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:

\n\n

http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html

\n\n

A common thread among the talks was the question: how to convert a research idea into a tool for software developers?

\n\n\n

In my mind, gradual typing is an answer to one instance of this question. The research idea is strong static type systems, and the software developers are the millions using dynamically typed languages. I know that static typing can make programs easier to write and maintain. The developers know that dynamic typing has benefits; moreover they know better than to migrate their code from one language to another on a whim. Gradual typing is a linguistic solution to the problem of adding the benefits of static typing to a dynamically typed language.

\n\n

Enough opinions, let’s talk about the talks.

\n\n

The morning session consisted of four talks:

\n\n\n\n

In the early afternoon, we had two talks on similar themes as the morning session:

\n\n\n\n

Next on the schedule were two talks about implementing advanced type systems in Racket’s macro expander (think: meta-level linguistic re-use, capture-avoiding substitution for free!)

\n\n\n\n

After a short break, we heard about something completely different:

\n\n\n\n

Last summer and fall, Jeremy Siek hosted two REUs (research experience for undergraduates) at Indiana University. The two students gave the next talks:

\n\n\n\n

Finally,

\n\n\n\n

This meeting was a great opportunity to reflect on the recent past and share opinions on what’s worth pursuing in the future. Many thanks to the participants, and to the NSF for the support!

\n\n
\n

If you want to know about the future, you need to ask the young people who will create it. Young people don’t know what can’t be done, and so they go ahead and do it. — Ivan Sutherland

")) ((? . 59) f post (u . "The Racket School 2018: Create your own language") (? . 59) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/04/27/the-racket-school-2018-create-your-own-language/index.html" . unix) (u . "/blog/2018/04/27/the-racket-school-2018-create-your-own-language/") (u . "2018-04-27T21:35:22") (? . 37) (? . 79) (c (u . "event") c (u . "Racket") c (u . "Author: Ben Greenman")) (u . "\n

The Racket School 2018: Create your own language • 9–13 July • Salt Lake City

") #t (u . "\n

The Racket School 2018: Create your own language • 9–13 July • Salt Lake City

\n\n\n

The Racket team has spent over thirty years developing and refining a coherent intellectual tradition for studying and building programming languages. This year’s school will introduce participants to Racket’s framework for language-oriented programming, which the summer school faculty recently spelled out in a a cover article in the Communications of the ACM

\n\n

Concretely, the 2018 Racket Summer School will cover the following topics:

\n\n\n\n

If these topics intrigue you, attend the Racket Summer School:

\n\n\n\n

This is not your run-of-the-mill summer school. We will do our best to make it exciting, entertaining, and useful to a broad spectrum of attendees, both academic and industrial.

\n\n

P.S. We will send you your first problem set in June, a month before the summer school to whet your appetite.

")) ((? . 28) f post (u . "Tutorial: Using Racket's FFI") (? . 28) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/06/27/tutorial-using-racket-s-ffi/index.html" . unix) (u . "/blog/2016/06/27/tutorial-using-racket-s-ffi/") (u . "2016-06-27T16:22:11") (? . 27) (? . 24) (c (u . "Racket") c (u . "FFI") c (u . "tutorial") c (u . "Author: Asumu Takikawa")) (u . "\n

Update: this post is now part of a series. Part 2 is\nhere\nand part 3 is\nhere.

\n\n

I’ve seen several people ask for a tutorial on Racket’s foreign\nfunction interface (FFI), which allows you to dynamically load\nC libraries for use in Racket code. While I think the\ndocumentation\nfor the FFI is quite good, it is a lot of information to process and\nthe overview examples may be tricky to run for a beginner.

\n\n

With that in mind, this blog post will provide a step-by-step tutorial\nfor Racket’s FFI that requires minimal setup. All that you will need to\nfollow along is a copy of Racket and ideally a DrRacket window.

") #t (u . "\n

Update: this post is now part of a series. Part 2 is\nhere\nand part 3 is\nhere.

\n\n

I’ve seen several people ask for a tutorial on Racket’s foreign\nfunction interface (FFI), which allows you to dynamically load\nC libraries for use in Racket code. While I think the\ndocumentation\nfor the FFI is quite good, it is a lot of information to process and\nthe overview examples may be tricky to run for a beginner.

\n\n

With that in mind, this blog post will provide a step-by-step tutorial\nfor Racket’s FFI that requires minimal setup. All that you will need to\nfollow along is a copy of Racket and ideally a DrRacket window.

\n\n\n

Before getting into the details, I wanted to note that the FFI library\nis based on the work of Eli Barzilay and Dmitry Orlovsky. They have\na Scheme Workshop paper\nthat you can read if you’re curious about the design.

\n\n

The tutorial will focus on using the Cairo\ngraphics library, mainly because it comes bundled with Racket.

\n\n

To start, let’s aim to reproduce the output of the \"multi segment caps\"\nC sample code on Cairo’s\nsamples page:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
cairo_move_to (cr, 50.0, 75.0);
cairo_line_to (cr, 200.0, 75.0);
 
cairo_move_to (cr, 50.0, 125.0);
cairo_line_to (cr, 200.0, 125.0);
 
cairo_move_to (cr, 50.0, 175.0);
cairo_line_to (cr, 200.0, 175.0);
 
cairo_set_line_width (cr, 30.0);
cairo_set_line_cap (cr, CAIRO_LINE_CAP_ROUND);
cairo_stroke (cr);
\n\n

In order to actually draw this example to the screen, we will need a Cairo\nsurface to draw on. So here is some boilerplate for you to execute before we\nactually play with the FFI:

\n\n
\n \n \n \n \n \n \n \n
> (require racket/draw)
> (define bt (make-bitmap 256 256))
> (define bt-surface (send bt get-handle))
\n\n

This uses the Racket drawing library racket/draw to construct\na bitmap object that we’ll draw on. The get-handle method just extracts a\nlow-level Cairo surface value that we can use.

\n\n

NB: these code snippets don’t come with a #lang declaration because\nthey simulate interactions at the REPL/interaction area in DrRacket.\nWhen following along, just copy & paste the snippets into your REPL.

\n\n

Our first real step is to import the FFI itself:

\n\n
\n \n \n \n
> (require ffi/unsafe)
\n\n

As the module name suggests, the FFI is unsafe and can cause your Racket process\nto segfault. If you’re following along in DrRacket, you will want to save your file\nfrequently.

\n\n

Next, we can load the Cairo library to obtain a foreign-library value, which\nis a handle that we use to access C values and functions:

\n\n
\n \n \n \n
> (define cairo-lib (ffi-lib #f))
\n\n

Since Cairo has already been loaded by the Racket process because of the\nracket/gui import earlier, we can supply #f here as an\nargument to ffi-lib. Normally you supply the name of a shared\nlibrary file such as \"libcairo\":

\n\n
\n

(define cairo-lib (ffi-lib \"libcairo\" '(\"2\" #f)))

\n\n

The last list argument specifies the accepted versions (#f allows\na version-less library). For this post, those details aren’t important but\nsee the docs on ffi-lib if you’re curious.

\n\n

Extracting functions

\n\n

Since the Racket FFI is a dynamic interface, we can pull out C functions\nat run-time using the get-ffi-obj function. The get-ffi-obj\nfunction takes three arguments:

\n\n\n\n

C types are a crucial concept for the FFI. They range from relatively\nsimple types like _int and _pointer to more complicated\ntype constructors such as _enum and _fun. As you probably noticed,\nC types are prefixed with an underscore by convention. You can also define\nyour own types by calling make-ctype with two functions that\nhandle marshalling between C and Racket code.

\n\n

To make progress with our Cairo code, we need to create a drawing context from\nthe surface object bt-surface that we defined a while ago. The\nrelevant function in the Cairo docs is\ncairo_create,\nwhich has the following type signature:

\n\n
\n \n \n \n \n \n
/* NB: this is C code */
cairo_t * cairo_create (cairo_surface_t *target);
\n\n

\n\n
To use this function from Racket, we will need to create a C type that describes\nits behavior. As you can see, the function takes a pointer to a cairo_surface_t and\nreturns a pointer to a cairo_t. Let’s start with a very simple C type\nthat matches up with this behavior:
\n\n
\n
\n

(_fun _pointer -> _pointer)

\n\n
This type provides very little safety (in particular, it lets you mix up different\nkinds of pointers), but it will work as a first step.\nNote that the FFI library uses infix arrow notation for its _fun type.
\n\n

The following definition shows how to use this type to obtain a foreign\nfunction:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n
> (define cairo-create
    (get-ffi-obj \"cairo_create\" cairo-lib
                 (_fun _pointer -> _pointer)))
\n\n

Then we can use cairo-create as an ordinary racket function:

\n\n
\n \n \n \n \n \n \n \n
> (define ctx (cairo-create bt-surface))
> ctx
\n

#<cpointer>

\n\n

Interlude: more type safety

\n\n

Before we move on to completing the Cairo sample, lets consider the safety of the\nC type we used again. Since we only specified _pointer types, it is easy\nto accidentally misuse the function:

\n\n
\n \n \n \n \n \n \n \n
; You may not want to actually run this
; a cairo_t is not a cairo_surface_t
(cairo-create (cairo-create bt-surface))
\n\n

To prevent such bad uses, it is good practice to use tagged pointer types\nusing define-cpointer-type. Here are two example definitions that\ncorrespond to the cairo_t and cairo_surface_t types from earlier:

\n\n
\n \n \n \n \n \n \n \n
; The leading underscores are mandatory
> (define-cpointer-type _cairo_t)
> (define-cpointer-type _cairo_surface_t)
\n\n

We can then redefine cairo-create with a better type, which will\nprevent ill-typed calls:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n
\n \n \n \n \n \n \n \n
> (define cairo-create
    (get-ffi-obj \"cairo_create\" cairo-lib
                 (_fun _cairo_surface_t -> _cairo_t)))
> (cairo-create (cairo-create bt-surface))
\n

cairo_surface_t->C: argument is not non-null

\n

`cairo_surface_t' pointer

\n

  argument: #<cpointer:cairo_t>

\n\n

Unfortunately our old definition of ctx doesn’t have this tag:

\n\n
\n \n \n \n \n \n
> (cpointer-has-tag? ctx 'cairo_t)
\n

#f

\n\n

Which means we will see errors if we try to use it in future interactions with\nthe more precise C type. To get around this, it’s also possible to update\nexisting pointers with a tag like this:

\n\n
\n \n \n \n \n \n \n \n
> (cpointer-push-tag! ctx 'cairo_t)
> (cpointer-has-tag? ctx 'cairo_t)
\n

#t

\n\n

Executing the tag push above is necessary to get some of the following snippets\nto work (if you are following along step-by-step).

\n\n

Macros for reducing boilerplate

\n\n

Now let’s start building the FFI bindings for the functions in the Cairo sample.\nFirst, let’s go ahead and look at all of the types for the sample functions\nfrom the C API docs:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n
void cairo_move_to (cairo_t *cr, double x, double y);
void cairo_line_to (cairo_t *cr, double x, double y);
void cairo_set_line_width (cairo_t *cr, double width);
void cairo_set_line_cap (cairo_t *cr, cairo_line_cap_t line_cap);
void cairo_stroke (cairo_t *cr);
\n\n

Starting with cairo_move_to, we can set up a definition like we did\nwith cairo_create before:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n
> (define cairo-move-to
    (get-ffi-obj
     \"cairo_move_to\"
     cairo-lib
     (_fun _pointer _double _double -> _void)))
\n\n

This starts to look awfully verbose once you start writing more of these\ndefinitions. Luckily, the FFI library comes with some definition forms\nin the ffi/unsafe/define library that help reduce the\nverbosity. Here’s an alternative definition of cairo-move-to\nusing the define-ffi-definer form from\nffi/unsafe/define.

\n\n
\n \n \n \n \n \n \n \n
> (require ffi/unsafe/define)
> (define-ffi-definer define-cairo cairo-lib)
\n \n \n \n \n \n \n \n
> (define-cairo cairo-move-to
    (_fun _cairo_t _double _double -> _void)
    #:c-id cairo_move_to)
\n\n

As you can see, the define-ffi-definer form lets you define a\nnew macro that lets you avoid writing the library value over and over.\nIf you stick to using C-style identifiers with underscores (e.g.,\ncairo_move_to) you also don’t need to supply the C name either.

\n\n

The definitions for cairo_line_to, cairo_set_line_width, and\ncairo_stroke aren’t very interesting, so I’ll just include them\nbelow without comment:

\n\n
\n \n \n \n \n \n \n \n
\n \n \n \n \n \n \n \n
> (define-cairo cairo-line-to
    (_fun _cairo_t _double _double -> _void)
    #:c-id cairo_line_to)
\n \n \n \n \n \n \n \n
> (define-cairo cairo-set-line-width
    (_fun _cairo_t _double -> _void)
    #:c-id cairo_set_line_width)
\n \n \n \n \n \n \n \n
> (define-cairo cairo-stroke
    (_fun _cairo_t -> _void)
    #:c-id cairo_stroke)
\n\n

The cairo_set_line_cap case is more interesting because the type\ncairo_line_cap_t is a C enumeration type. Racket’s FFI comes with\nconvenience forms for defining enumeration types—\n though it’s possible\nto encode them yourself too. The general philosophy of the Racket FFI\nis to keep the C parts to a minimum and let you build abstractions\nin Racket libraries. Here’s a quote from the Barzilay and Orlovsky\npaper on that:

\n\n
\n

Our design follows a simple principle: keep C-level\nfunctionality to a minimum.

\n\n

and specifically about enumerations:

\n\n
\n

For example, the C level part of our interface does not commit to a\nspecific implementation for enumerations — it simply exposes C integers.

\n\n

To define an enumeration, we can use the _enum form. This procedure\nsets up a new C type which converts between Racket symbols and the underlying\ninteger representations. For the cairo_line_cap_t type, it suffices to\njust supply the cases as a list of symbols:

\n\n
\n \n \n \n
\n \n \n \n \n \n
> (define _cairo_line_cap_t
    (_enum '(butt round square)))
\n\n

The exact symbols that we specify are not important, since they just map to\nintegers anyway. The choice depends on what is convenient for the Racket interface.\nIt’s also possible to specify how the symbols map to integers more precisely\n(see the docs on _enum for those details).

\n\n

Given this type, we can specify the type for the line cap function:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n
> (define-cairo cairo-set-line-cap
    (_fun _cairo_t _cairo_line_cap_t -> _void)
    #:c-id cairo_set_line_cap)
\n\n

Putting it all together

\n\n

Now that we have foreign function definitions for all of the relevant procedures,\nwe can just transcribe the example from the beginning into Racket syntax:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
(cairo-move-to ctx 50.0 75.0)
(cairo-line-to ctx 200.0 75.0)
 
(cairo-move-to ctx 50.0 125.0)
(cairo-line-to ctx 200.0 125.0)
 
(cairo-move-to ctx 50.0 175.0)
(cairo-line-to ctx 200.0 175.0)
 
(cairo-set-line-width ctx 30.0)
(cairo-set-line-cap ctx 'round)
(cairo-stroke ctx)
\n\n

Executing these procedure calls will draw into the Cairo surface we set up earlier,\nwhich is connected to our original Racket bitmap object bt. To see the\nresults of what we drew, we can just evaluate bt at the REPL. But it’s\na little nicer if we use the pict library to draw a frame around\nit to distinguish it from the background:

\n\n
\n \n \n \n \n \n \n \n
> (require pict)
> (linewidth 2 (frame (bitmap bt)))
\n

\"image\"

\n\n

And we’re done! Of course, there is a lot more to the FFI. For example, I haven’t\ncovered how to handle C functions that return multiple results through pointer\narguments. Or how to interoperate between Racket and C structs. I’m hoping to\ncover these in a future blog post, but in the meantime happy FFI hacking!

")) ((? . 60) f post (u . "[Toward Type-Preserving Compilation of Coq, at POPL17 SRC (cross-post)](https://williamjbowman.com/blog/2017/01/03/toward-type-preserving-compilation-of-coq-at-popl17-src/)") (? . 60) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/01/03/-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src/index.html" . unix) (u . "/blog/2017/01/03/-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src/") (u . "2017-01-03T15:15:57") (? . 61) (? . 62) (c (u . "Author: William J. Bowman")) (? . 5) #f (? . 5)) ((? . 63) f post (u . "Java and Migratory Typing") (? . 63) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/12/02/java-and-migratory-typing/index.html" . unix) (u . "/blog/2018/12/02/java-and-migratory-typing/") (u . "2018-12-02T14:41:53") (? . 33) (? . 77) (c (u . "migratory typing") c (u . "java") c (u . "transient") c (u . "Author: Ben Greenman")) (u . "\n

The transient approach to migratory typing (circa 2014) is similar to type erasure in Java (circa 2004) in a few interesting ways.

") #t (u . "\n

The transient approach to migratory typing (circa 2014) is similar to type erasure in Java (circa 2004) in a few interesting ways.

\n\n\n

Migratory typing

\n\n

The goal of migratory typing is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:

\n\n\n\n

There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.

\n\n

A typical migratory typing system adds a static type checker to a dynamically typed language (examples), but one could also extend the type system of a statically-typed language; for example, by adding dependent types. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.

\n\n

Java’s implementation of migratory typing has some interesting things in common with the transient implementation strategy recently proposed by Michael Vitousek and collaborators (DLS’14, POPL’17). The goal of this post is to demonstrate the connections.

\n\n

Erasure migratory typing

\n\n

Before we compare Java 1.5.0 to transient, let’s review a simpler strategy: the erasure approach to migratory typing.

\n\n

TypeScript is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.

\n\n
\n

The DefinitelyTyped repository provides TypeScript type definitions for many JavaScript libraries.

\n\n

The TypeScript compiler erases types; every type T in the source code translates to the universal “JavaScript type”. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:

\n\n
(function (n0 : number, n1 : number) { return n0 + n1; })\n\n// ==(compiles to)==>\n\n(function (n0, n1) { return n0 + n1; })
\n\n

TypeScript satisfies goals G1 and G3 for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal G2 because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable n0 is not guaranteed to point to a number at run-time. On one hand, this means the type annotations have no effect on the behavior of a program — and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.

\n\n

In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:

\n\n\n\n

Transient migratory typing

\n\n

Reticulated is a migratory typing system for Python that follows a transient implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal G1, the compilation to Python provides interoperability (goal G3), and the assertions partially meet goal G2.

\n\n
\n

These certain inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of near-constant, see On the Cost of Type-Tag Soundness, section 2.

\n\n

For example, here is a Reticulated function that computes the average of a list of numbers:

\n\n
# Reticulated (commit e478343)\ndef average(nums : List(Float)) -> Float:\n  if ns:\n    return sum(ns) / len(ns)\n  else:\n    raise ValueError(\"average: expected non-empty list\")
\n\n

and here is the Python code it compiles to:

\n\n
from retic.runtime import *\nfrom retic.transient import *\nfrom retic.typing import *\n\ndef average(nums):\n    check_type_list(nums)\n    if ns:\n        return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns)))\n    else:\n        raise check_type_function(ValueError)('average: expected non-empty list')
\n\n
\n

Note: the Reticulated syntax for type annotations is similar to the one proposed in PEP 484, but not identical. For example, Reticulated does not require forward references to be embedded in strings.

\n\n

The Reticulated compiler removes all type annotations and inserts check_type assertions throughout the code. In average, these assertions check that: (1) the input is a list, (2) the output is a float, (3) and the names sum len and ValueError point to callable values. That’s all. The assertions do not check that nums contains only floating-point numbers.

\n\n
\n

The assertions also do not check that the function bound to sum is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.

\n\n

If nums contains something other than floating point numbers, then the call to average may cause sum to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of sum in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.

\n\n

Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor C(T), e.g.:

\n\n
  C(Float)                = Float\n  C(List(Float))          = List\n  C(List(Float) -> Float) = ->
\n\n

Consequently, Reticulated has a slightly stronger run-time guarantee than Python:

\n\n\n\n

Java migratory typing

\n\n

Java 1.5.0 added generic types to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.

\n\n

Review: generic types

\n\n

Suppose we want to write a Box class that holds some kind of value; the value could be an Integer or a String or anything else. Here is a pre-generics definition:

\n\n
class Box {\n  private Object val;\n\n  public Box(Object val) { this.set(val); }\n\n  public void set(Object val) { this.val = val; }\n\n  public Object get() { return this.val; }\n}
\n\n

With this definition is it possible to make boxes that hold different types of values:

\n\n
// good!\nBox iBox = new Box(new Integer(4));\nBox sBox = new Box(new String(\"X\"));
\n\n

but it is also possible to “change the type” of the contents of a Box:

\n\n
// maybe bad!\niBox.set(new String(\"not a number\"));
\n\n

and some calls to get must be followed by a type cast:

\n\n
// annoying!\n((String) sBox.get()).charAt(0);
\n\n
\n\n

With generics, we can give a name (e.g. ValType) to “the type of the value inside a box”:

\n\n
class GBox<ValType> {\n  private ValType val;\n\n  public GBox(ValType val) { this.set(val); }\n\n  public void set(ValType val) { this.val = val; }\n\n  public ValType get() { return this.val; }\n}
\n\n

and now we can tell the type checker to check different boxes differently (satisfying goal G1):

\n\n
GBox<Integer> iBox = new GBox<Integer>(new Integer(0));\nGBox<String> sBox = new GBox<String>(new String(\"A\"));\n\n// iBox.set(new String(\"not a number\")); // Type Error, good!\n\nsBox.get().charAt(0); // no cast, good!
\n\n

Backwards compatibility & danger

\n\n

Java generics are backwards-compatible with older code (goal G3). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a GBox:

\n\n
// Java 1.4.0 method\npublic static void evil(GBox b) { b.set(666); }\n\n// Java 1.5.0 method\npublic static void test() {\n  GBox<String> sBox = new GBox<String>(new String(\"A\"));\n  evil(sBox); // OK, but generates unchecked warning\n  sBox.get().charAt(0);\n}
\n\n

The code above passes the type checker (with a warning about the evil method), and so it seems as though running the code will run the nonsense method call 666.charAt(0) and lead to evil behavior. The actual result, however, is a cast error immediately after the call sBox.get() returns.

\n\n

Based on the cast error, we can tell that the compiler does not trust the type GBox<String> and inserts a run-time check that the result of the .get() is a string object.

\n\n
\n

“Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.” Generics in the Java Programming Language, Section 6.1

\n\n

Java Type Erasure

\n\n

In order to support pre-generics and post-generics code on the same virtual machine, the Java compiler erases generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the String in GBox<String> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via occurrence typing.)

\n\n

After erasure, the GBox<ValType> class declaration loses its parameter:

\n\n
// Erase `ValType`, replace with `Object`\nclass GBox {\n  private Object val;\n\n  public GBox(Object val) { this.set(val); }\n\n  public void set(Object val) { this.val = val; }\n\n  public Object get() { return this.val; }\n}
\n\n

and the client code gains a cast:

\n\n
GBox sBox = new GBox(new String(\"A\"));\n\n((String) sBox.get()).charAt(0);
\n\n

So far, so good. But it’s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following “natural” definition of an ArrayList is not permitted:

\n\n
class ArrayList<T> {\n  private T[] data;\n  private int size;\n\n  public ArrayList(int capacity) {\n    data = new T[capacity];\n    size = 0;\n  }\n\n  public T get(int ix) {\n    // TODO bounds check\n    return data[ix]\n  }\n\n  // ....\n}
\n\n

The trouble is that T does not say anything about the data that a new array needs to handle:

\n\n
ArrayList.java:6: error: generic array creation\n    data = new T[capacity];
\n\n

The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:

\n\n
  // possibly dangerous, if `data` is aliased to an `Object[]`\n  public ArrayList(int capacity) {\n    data = (T[]) new Object[capacity];\n    size = 0;\n  }
\n\n

The other is to unsafely cast array elements in the get method, and elsewhere:

\n\n
class ArrayList<T> {\n  private Object[] data;\n  private int size;\n\n  public ArrayList(int capacity) {\n    data = new Object[capacity];\n    size = 0;\n  }\n\n  public T get(int ix) {\n    boundsCheck(ix);\n    return (T) data[ix];\n  }\n\n  // ....\n}
\n\n

Both may potentially lead to heap pollution.

\n\n
\n

\"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.

\n

\"Ultimately, the most important motivation for this decision is compatibility with existing code.\" Java Language Specification, section 4.7

\n\n

Run-time guarantees

\n\n

By contrast to Reticulated’s C(T) transformation, the following G(T) transformation describes generic-type erasure, where T<T1> describes a type T with parameter T1 and A[T1, T2] describes a type variable A with lower bound T1 and upper bound T2:

\n\n
  G(T<T1>)     = G(T)\n  G(A[T1, T2]) = G(T1)\n  G(T)         = T      otherwise
\n\n

If generic-type erasure results in a type mismatch (e.g., in sBox.get().charAt(0) above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:

\n\n\n\n

Discussion

\n\n

TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name migratory typing describes this kind of language extension.

\n\n
\n

Gradual typing is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (example).

\n\n

The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript libraries, however, can enforce some types.)

\n\n
\n

TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is MACLISP. For an erasure manifesto, see Pluggable Type Systems.

\n\n

The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (POPL 2017). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The Pyret language has been successful with comparable shallow checks.

\n\n
\n

Note: the POPL 2017 paper advertises an “open-world soundness”, but I do not see how this idea is different from the older idea of soundness in a multi-language system (TOPLAS 2009, DLS 2006).

\n\n

Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety — enough to prevent the use of a generic object from corrupting the state of a VM instance.

\n\n

Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (example 1, example 2). The C# language has a similar type system and enforces generics at run-time (sources: blog post, PLDI 2001 paper, backup link to paper)

\n\n

Acknowledgments

\n\n

Thank you to Ryan Culpepper and Jesse Tov for noticing the similarity between Java’s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.

")) ((? . 64) f post (u . "PRL Offsite 2019 Retrospective") (? . 64) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/12/12/prl-offsite-2019-retrospective/index.html" . unix) (u . "/blog/2019/12/12/prl-offsite-2019-retrospective/") (u . "2019-12-12T12:51:53") (? . 70) (? . 73) (c (u . "offsite") c (u . "Author: Ben Greenman") c (u . "Author: Olek Gierczak")) (u . "\n

On November 11th 2019, the PRL had a private offsite meeting at the More Than Words bookstore in downtown Boston. For future offsite organizers, this post records what happened and how.

") #t (u . "\n

On November 11th 2019, the PRL had a private offsite meeting at the More Than Words bookstore in downtown Boston. For future offsite organizers, this post records what happened and how.

\n\n\n

Early Planning, Goals

\n\n

Every Fall, the PRL holds a kickoff meeting to assign roles for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.

\n\n
\n

Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the Warren Conference Center is Northeastern University’s only franchised off-campus venue.

\n\n

The motivation for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students — both to get to know the group, and to become known. They’re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the Spring 2017 edition of HOPL.

\n\n

Our primary goal this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.

\n\n

In light of the main goal, we prefer to call the event an “offsite” (or “offsite meeting”) rather than a “retreat” because the target was an informative day rather than a relaxing one.

\n\n

Booking and Logistics

\n\n

Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that’s all we had.

\n\n

Date = Nov 11

\n\n

In the beginning, we chose Veterans’ Day (2019–11–11) and Northeastern reading day (2019–12–05) as possible dates. We ended up with Veterans’ Day.

\n\n

A small number of lab members were opposed to Veterans’ Day. They gave two reasons: the Fall semester is especially busy, and Veterans’ Day is a federal holiday.

\n\n

Venue = MTW

\n\n

The first venue we tried was the Warren Conference Center in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the Khoury Events Team to reach out on our behalf. They returned with an email address and event menu (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.

\n\n\n

Second, we contacted the Northeastern University Alumni Center. They were not open on Veterans’ Day 2019.

\n\n

Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: More Than Words.

\n\n

More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!

\n\n
\n

More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you’ll see a mix of “employees” and “volunteers” helping out.

\n\n

Booking was complicated, though, by the fact that Northeastern requires liability insurance for all off-campus events. If you are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.

\n\n

Here are more facts about MTW:

\n\n\n\n

Keynote = None

\n\n

The original, ambitious plan was to invite two keynote speakers — one working in industry and one from academia — to enrich the offsite with new knowledge. And because this was the PRL’s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)

\n\n

We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.

\n\n

Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).

\n\n\n

Food, Coffee, Tea

\n\n

Flour Bakery + Cafe provided lunch. For most days, you can order from Flour using an online form. For holidays, you may need to send an email — that’s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan & gluten-free everything spiced salads.

\n\n

Cuppacoffee provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything — the order + napkins, milk, cream cheese, butter, and knives — ourselves because Cuppacoffee is very close to MTW.

\n\n

CMart sold us water and juices. (There is also a Whole Foods near MTW.)

\n\n

At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour’s sticky buns.

\n\n

Event Program

\n\n

The following assumptions/goals constrained the schedule for the day:

\n\n\n\n

We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.

\n\n

The activity was codewalks. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the IntegerDivider class from a silly FizzBuzz implementation and a solution to the Dutch national flag problem). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and egoless programming.

\n\n

The talks ran conference-style. One organizer played “session chair” to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.

\n\n

For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.

\n\n

Looking Back at the Details

\n\n

Reading Day in the Spring may be a good, free date for future retreats.

\n\n

We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.

\n\n

The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we’d keep the morning breaks short — just enough to use the restroom and grab a coffee — and aim for 30-min breaks in the afternoon.

\n\n

Groups of three 15-minute talks worked well, but groups of four talks might be equally good.

\n\n

Perhaps each speaker should get the chance to pick a talk length. NEPLS, for example, allows a choice between 5-min. and 30-min. talks.

\n\n

The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick “topics” beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.

\n\n\n

Looking Back, Overall

\n\n

This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.

\n\n

There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time — keep the same date & program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.

\n\n

With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.

")) ((? . 53) f post (u . "Tutorial: Racket FFI, part 3") (? . 53) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/07/11/tutorial-racket-ffi-part-3/index.html" . unix) (u . "/blog/2016/07/11/tutorial-racket-ffi-part-3/") (u . "2016-07-11T17:33:40") (? . 24) (? . 52) (c (u . "Racket") c (u . "FFI") c (u . "tutorial") c (u . "Author: Asumu Takikawa")) (u . "\n

This is part 3 of my tutorial for using the Racket FFI. You can find part 1\nhere\nand part 2\nhere.

\n\n

In this post, we will experiment with some low-level operations with pointers,\nunion types, and custom C types. The main takeaway will be the custom C types,\nwhich let you define abstractions that hide the details of the C representation\nwhen manipulating data in Racket.

") #t (u . "\n

This is part 3 of my tutorial for using the Racket FFI. You can find part 1\nhere\nand part 2\nhere.

\n\n

In this post, we will experiment with some low-level operations with pointers,\nunion types, and custom C types. The main takeaway will be the custom C types,\nwhich let you define abstractions that hide the details of the C representation\nwhen manipulating data in Racket.

\n\n\n

As in the second post, let’s start with some prologue code that establishes\nthe definitions from the previous two posts. But first, I’m getting tired of\nwriting the #:c-id identifier notation for the underscored C function\nnames.

\n\n

Instead, let’s use a third-party package that I wrote that lets you avoid the\nboilerplate. To install the package, you can either invoke the following\nincantation in a command-line:

\n\n
\n \n \n \n
$ raco pkg install ffi-definer-convention
\n\n

or you can just execute the following snippet in Racket:

\n\n
\n \n \n \n \n \n \n \n
> (require pkg)
> (pkg-install-command #:skip-installed #t \"ffi-definer-convention\")
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
\n

Resolving \"ffi-definer-convention\" via https://download.racket-lang.org/releases/7.9/catalog/

\n

Resolving \"ffi-definer-convention\" via https://pkgs.racket-lang.org

\n

Downloading repository git://github.com/takikawa/racket-ffi-definer-convention

\n

raco setup: version: 7.9

\n

raco setup: platform: x86_64-linux [3m]

\n

raco setup: target machine: racket

\n

raco setup: installation name: 7.9

\n

raco setup: variants: 3m

\n

raco setup: main collects: /usr/share/racket/collects

\n

raco setup: collects paths:

\n

raco setup:   /usr/share/racket/collects

\n

raco setup: main pkgs: /usr/share/racket/pkgs

\n

raco setup: pkgs paths:

\n

raco setup:   /usr/share/racket/pkgs

\n

raco setup:   /root/.local/share/racket/7.9/pkgs

\n

raco setup: links files:

\n

raco setup:   /usr/share/racket/links.rktd

\n

raco setup:   /root/.local/share/racket/7.9/links.rktd

\n

raco setup: main docs: /usr/share/racket/doc

\n

raco setup: --- updating info-domain tables ---                    [22:20:56]

\n

raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd

\n

raco setup: --- pre-installing collections ---                     [22:20:56]

\n

raco setup: --- installing foreign libraries ---                   [22:20:56]

\n

raco setup: --- installing shared files ---                        [22:20:56]

\n

raco setup: --- compiling collections ---                          [22:20:56]

\n

raco setup: --- parallel build using 4 jobs ---                    [22:20:56]

\n

raco setup: 3 making: <pkgs>/ffi-definer-convention

\n

raco setup: --- creating launchers ---                             [22:20:57]

\n

raco setup: --- installing man pages ---                           [22:20:57]

\n

raco setup: --- building documentation ---                         [22:20:57]

\n

raco setup: 3 running: <pkgs>/ffi-definer-convention/ffi-definer-convention.scrbl

\n

raco setup: 3 rendering: <pkgs>/ffi-definer-convention/ffi-definer-convention.scrbl

\n

raco setup: 2 rendering: <pkgs>/racket-index/scribblings/main/user/local-redirect.scrbl

\n

raco setup: 1 rendering: <pkgs>/racket-index/scribblings/main/user/release.scrbl

\n

raco setup: 0 rendering: <pkgs>/racket-index/scribblings/main/user/search.scrbl

\n

raco setup: 0 rendering: <pkgs>/racket-index/scribblings/main/user/start.scrbl

\n

raco setup: --- installing collections ---                         [22:21:03]

\n

raco setup: --- post-installing collections ---                    [22:21:03]

\n\n

This will install the package and compile its contents. If you’re curious, the\ndocs for the package are available\nhere.

\n\n

Note: if you’ve never installed a package before, you may want to glance\nat the\npackage system docs.\nA tl;dr of packages is that they bundle Racket collections, which are\nsets of modules that you can refer to in a location-independent fashion such as\npict or racket/list.

\n\n

Anyhow, here’s the prologue code:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
#lang racket
(require racket/draw
         ffi/unsafe
         ; avoid conflict with below
         (except-in ffi/unsafe/define
                    define-ffi-definer)
         ; the new 3rd-party pkg
         ffi-definer-convention
         pict)
 
; C types
(define-cpointer-type _cairo_t)
(define-cpointer-type _cairo_surface_t)
(define _cairo_line_cap_t
  (_enum '(butt round square)))
 
(define cairo-lib (ffi-lib #f))
(define-ffi-definer define-cairo cairo-lib
  ; describes how to transform from
  ; Racket to C ids
  #:make-c-id convention:hyphen->underscore)
 
; the foreign functions
; note lack of #:c-id keyword arguments
(define-cairo cairo-create
  (_fun _cairo_surface_t -> _cairo_t))
(define-cairo cairo-move-to
  (_fun _cairo_t _double _double -> _void))
(define-cairo cairo-line-to
  (_fun _cairo_t _double _double -> _void))
(define-cairo cairo-set-line-width
  (_fun _cairo_t _double -> _void))
(define-cairo cairo-stroke
  (_fun _cairo_t -> _void))
(define-cairo cairo-set-line-cap
  (_fun _cairo_t _cairo_line_cap_t -> _void))
 
; (_cairo_t -> Void) -> Pict
; do some drawing and give us the pict
(define (do-cairo f)
  (define bt (make-bitmap 256 256))
  (define bt-surface (send bt get-handle))
  (f (cairo-create bt-surface))
  (linewidth 2 (frame (bitmap bt))))
\n\n

Notice that the define-cairo forms don’t have any #:c-id keywords\nanymore. Instead, the prologue code uses an overriden define-ffi-definer\nfrom my package that supports a #:make-c-id keyword that lets you specify\na naming convention to follow.

\n\n

Also, instead of creating a single bitmap and drawing into it, we now have a\ndo-cairo function that takes a drawing function. When called,\ndo-cairo will call the given function with a new bitmap object and\nreturn the result.

\n\n

Now let’s get to the main point of this blog post. Let’s say that we want to play\nwith Cairo path\nobjects this time. A path is\ndefined\nas a struct with the following structure:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n
typedef struct {
    cairo_status_t status;
    cairo_path_data_t *data;
    int num_data;
} cairo_path_t;
\n\n

To manipulate paths, we want to define a FFI C type that corresponds to this\nstruct definition. But before that, it’s\nuseful to define C types for the types of values in the path struct’s fields. First,\nlet’s specify that a cairo_status_t is an integer type:

\n\n
\n \n \n \n
> (define _cairo_status_t _int)
\n\n

It’s actually an enum, but for the examples in this post we don’t care about\ndistinguishing different statuses. Next, the data field of a path struct is an\narray of\npath data objects.\nEach path data object is a cairo_path_data_t,\nwhich is specified with a C union:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
union _cairo_path_data_t {
    struct {
        cairo_path_data_type_t type;
        int length;
    } header;
    struct {
        double x, y;
    } point;
};
\n\n

Helpfully, the FFI library comes with support for unions with the\n_union type constructor. The constructor takes arbitrarily\nmany arguments, one for each sub-case in the union. It’s pretty\nstraightforward to specify this type too:

\n\n
\n \n \n \n \n \n \n \n
; the path data type is just an enum
\n \n \n \n \n \n
> (define _cairo_path_data_type_t
    (_enum '(move-to line-to curve-to close-path)))
\n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define _cairo_path_data_t
    (_union ; the header case
            (_list-struct _cairo_path_data_type_t
                          _int)
            ; the point case
            (_list-struct _double _double)))
\n\n

There’s a new type constructor here so let me explain that first.\nThe _list-struct constructor translates between a C struct\nand a fixed-length list of C objects on the Racket side. Unlike\ndefine-cstruct, this constructor doesn’t define any selectors\nor anything like that. Instead, you can manipulate the struct as an\nordinary list.

\n\n

Each of the path data structs in the path data array will be manipulated with\nthe _cairo_path_data_t type. Union types are a bit cumbersome unfortunately\nbecause the programmer has to distinguish the cases in the union manually\non the Racket-side. Let me illustrate this with some code:

\n\n
\n \n \n \n \n \n \n \n \n \n
; create a union from a list of doubles
\n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define a-union-val
    (cast (list 1.3 5.8)
          ; source type
          (_list-struct _double _double)
          ; target type
          _cairo_path_data_t))
> a-union-val
\n

#<union>

\n\n

This snippet first construct a union object (via the _cairo_path_data_t\ntype) using a cast. A cast is an operation that lets you coerce\nfrom one C type to another. We use it in this example since it’s an easy way\nto generate a union object.

\n\n

The second line shows that a union prints as an opaque object. You can’t do\nanything with a union in Racket unless you project it to one of the sub-cases with\nthe union-ref function.\nThis projection is unsafe, in the sense that if you don’t know which of\nthe sub-cases in the union is the correct one, you will get potentially non-sensical\ndata out of the union.

\n\n

More concretely, let’s see what happens if we try to extract a value out of the\nunion both correctly and incorrectly:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
; correct (matches construction)
; cases are zero-indexed and ordered as written
> (union-ref a-union-val 1)
\n

'(1.3 5.8)

; incorrect, error
> (union-ref a-union-val 0)
\n

enum:int->_cairo_path_data_type_t: expected a known

\n

#<ctype:ufixint>, got: 3435973837

\n\n

Note that in the incorrect case we get an error saying that the FFI failed\nto convert the C value to a Racket value following the given C type. We were\nlucky in this case, but in general you can have silent failures where the\ndata is nonsense.

\n\n

With union types like these, there is usually some way to figure out which case\nof the union you are in. This may be accomplished in C using an extra struct\nfield or a variable that indicates the variant. Alternatively, there may be some\nset order that cases appear in data structures.

\n\n

With this Cairo API in particular, the position of the elements in the array\ntells you which of the union cases it’s in. The array always starts with a\nheader element, and then follows with some number of data elements (the exact\nnumber is determined by the type indicated in the header). We can therefore\nreference the appropriate element of the union based on this ordering.

\n\n

So before moving on, let’s recap: so far we have made C types that describe\nthe data elements in a cairo path with unions. Next we’ll figure out how to\ndeal with the array itself.

\n\n

Some low-level operations

\n\n

Since we still don’t have a C type for cairo_path_t, let’s go ahead\nand make a simple one where we punt on the work of specifying the array\ntype:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n \n \n
> (define _simple_cairo_path_t
    (_list-struct _cairo_status_t
                  _pointer
                  _int))
\n\n

In this type, we have specified the array as a bare _pointer.\nFor some added safety, we could also use something like\n(_cpointer 'cairo_status_t), which sets up a tagged pointer\ntype like we saw in the first blog post with\ndefine-cpointer-type.

\n\n

We’ve seen the _pointer type before, but haven’t actually done\nanything with values of those types except pass them around as arguments.\nIt turns out it is possible to do a bit more with pointers.

\n\n

Before we get to that, let’s go ahead and set up an FFI binding for\ncairo_copy_path so that we can obtain a path struct to manipulate:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n
\n \n \n \n \n \n
> (define-cairo cairo-copy-path
    (_fun _cairo_t -> _pointer))
> (define a-path #f)
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (do-cairo (λ (ctx)
              ; Do stuff to make the current
              ; path non-empty
              (cairo-move-to ctx 50.0 50.0)
              (cairo-line-to ctx 206.0 206.0)
              (cairo-move-to ctx 50.0 206.0)
              (cairo-line-to ctx 115.0 115.0)
              ; Get the current path
              (set! a-path (cairo-copy-path ctx))
              ; Stroke clears the path
              ; so do it last
              (cairo-stroke ctx)))
\n

\"image\"

> a-path
\n

#<cpointer>

\n\n

Note that cairo-copy-path gives us a pointer to a path struct\nrather than a path struct directly. Because of that, we need to know\nhow to manipulate pointers.\nThe most useful function for pointers is ptr-ref, which\nlets you dereference a pointer and access it at some concrete C type.

\n\n

Note: the ptr-ref function also takes an optional\noffset argument which we will be used in an example later.

\n\n

For example, we can use a-path as a _simple_cairo_path_t:

\n\n
\n \n \n \n \n \n \n \n
\n \n \n \n \n \n
> (define simple-path
    (ptr-ref a-path _simple_cairo_path_t))
> simple-path
\n

'(0 #<cpointer> 8)

\n\n

And now we have a Racket representation of the struct that the pointer\npoints to. Now notice that the data array field of the struct is also\na pointer as we specified earlier. To convert this to a more useful form,\nwe can use ptr-ref again with an array type:

\n\n
\n \n \n \n \n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define array
    (ptr-ref ; the pointer
             (second simple-path)
             (_array/list _cairo_path_data_t
                          ; length field
                          (third simple-path))))
> array
\n

'(#<union> #<union> #<union> #<union> #<union> #<union> #<union> #<union>)

\n\n

The elements of the array are all unions, as we would expect. This is\na bit annoying to use though. We have to know the structure of the\narray and reference the correct variant appropriately:

\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (union-ref (first array) 0)
\n

'(move-to 2)

> (union-ref (second array) 1)
\n

'(50.0 50.0)

; nonsense data here, wrong union case
> (union-ref (third array) 1)
\n

'(4.2439915824246e-314 0.0)

\n\n

One thing we could do is write a helper function that converts this array\ninto a more useful format. It would look at each header, and then consume\nthe number of data elements specified in the header element (e.g., 1\nin the example above because the length includes the header)\nand convert them appropriately.

\n\n

An alternative is to define a custom C type that handles all of\nthis conversion automatically for us, so that as a user of the Cairo\nFFI bindings we don’t need to think about applying helper functions and\ndereferencing pointers.

\n\n

Custom C types

\n\n

I briefly remarked on how to create custom C types in the first blog post,\nbut let me go over that again in more detail. A custom C type is constructed\nby providing a base C type to use along with two conversion functions.\nThe first function converts from a Racket value to a value that fits the\nbase C type. The second converts in the other direction from a value of\nthe base C type to a Racket value.

\n\n

In this way, it’s possible to conduct interesting conversions, such as\ndereferencing union objects automatically.

\n\n

Now let’s make a custom C type for Cairo paths that will represent the\ndata elements as a sequence in which each item in the sequence is a list\nwith an action symbol followed by the data elements for that action.

\n\n

First, we’ll start by defining a struct type for the Racket representation\nof Cairo paths:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n
> (struct cairo-path (ptr)
    #:property prop:sequence
    (λ (p) (in-cairo-path p)))
\n\n

The representation will store one field ptr which, as the name\nsuggests, will store a pointer value. We’ll see what to do with this\npointer later.

\n\n

This definition uses a structure type property to make instances of\ncairo-path automatically work as sequences. This means that you\ncan iterate over them with a for loop or apply sequence-ref\non them. The property takes a function that takes an instance of the struct\ntype itself (here p) and that returns a sequence.

\n\n

We’ll later define the in-cairo-path function that will actually\nconstruct the relevant sequence for us. For now, let’s see how to construct\nthe C type given this struct type:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define _cairo_path_t
    (let ()
      ; Extract pointer out of representation
      (define (racket->c rkt)
        (cairo-path-ptr rkt))
      ; Just apply the Racket constructor
      (define (c->racket cobj)
        (cairo-path cobj))
      (make-ctype _pointer
                  racket->c
                  c->racket)))
\n\n

The base type for this _cairo_path_t is a _pointer type. Since\nthe Cairo API returns pointers to new path values, it’s hard to avoid using some\nkind of pointer type as the base type here.

\n\n

This definition right-hand-side defines the two conversion functions between\nRacket and C. Both are very simple because of how we’ve set up the representation.\nIn the Racket to C case, we simply extract the pointer field of the struct. In\nthe other direction, we just stuff the pointer into a struct.

\n\n

The real work is done by the helper function that makes a\ncairo-path instance work as a sequence.

\n\n

Starting top-down, let’s look at the definition of in-cairo-path:

\n\n
\n \n \n \n \n \n
; Cairo-Path -> Sequence
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define (in-cairo-path path)
    (define pp (cairo-path-ptr path))
    (match-define
      (list _ array-ptr len)
      (ptr-ref pp _simple_cairo_path_t))
    (make-do-sequence
      (λ ()
        (values (pos->element array-ptr)
                (next-pos array-ptr)
                0
                (λ (pos) (< pos len))
                #f #f))))
\n\n

The first thing the function does is extract the pointer out of the\nrepresentation, and then immediately calls ptr-ref on it. This\nlets us manipulate the C path struct using the simple representation we\ndefined in the first part of the blog post.

\n\n

Note: in case you’re not very familiar with Racket pattern matching, the\nmatch-define form lets you define potentially multiple variables\nusing a pattern, similar to Haskell or OCaml’s let statement.\nThe first argument clause\nis a pattern and the second is an expression to match on. Check it out\nin the\ndocs.

\n\n

After extracting the array pointer and the array length from the\npath value, we pass them onto some helper functions that define the\nsequence. The usual way to define a new kind of sequence is to use the\nmake-do-sequence function. Essentially, make-do-sequence\ntakes a bunch of arguments that specify how to get the an element of\na sequence, how to advance a sequence, how to start, and how to end\nthe sequence.

\n\n

Note: technically make-do-sequence actually takes a thunk which\nproduces a number of values. These values are effectively like arguments\nthough. The reason why it’s a thunk is that you may wish to\nrun some initialization code that runs when the sequence is started\n(e.g., imagine opening a network connection), and your sequence functions\n(like advancing the sequence) may depend on the result of that\ninitialization code.

\n\n

In our case, we supply some curried functions that can extract elements\nout of the underlying C array. Here is the pos->element function\nand its helpers:

\n\n
\n \n \n \n \n \n \n \n \n \n
; CPointer -> Integer -> Element
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define ((pos->element ptr) pos)
    ; Extract the data path header
    (define header
      (union-ref
       (ptr-ref ptr _cairo_path_data_t pos)
       0))
    (define type   (first header))
    ; Length includes header, so subtract 1
    (define len    (sub1 (second header)))
    (define pos*   (add1 pos))
    (define points (get-points ptr pos* len))
    (cons type points))
; CPointer Integer Integer -> (Listof Data)
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define (get-points ptr pos num-points)
    (for/list ([i (in-range num-points)])
      (union-ref (ptr-ref ptr
                          _cairo_path_data_t
                          ; offset argument
                          (+ pos i))
                 1)))
\n\n

This code encodes the API usage protocol that Cairo specifies, where each\nheader element in the path is followed by some number of data elements.\nEach header specifies the length, so we can loop in get-points\nfrom the position after the header until we reach the given length. At\neach point, we dereference the appropriate union element.

\n\n

Advancing the sequence is simpler, since all we need to do is some arithmetic\non the length given by header elements:

\n\n
\n \n \n \n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (define ((next-pos ptr) pos)
    (define header
      (union-ref
       (ptr-ref ptr _cairo_path_data_t pos)
       0))
    (define len (second header))
    (+ len pos))
\n\n

Note that determining the end of the sequence is very easy. It’s just\na matter of comparing the current position to the total length given in the\npath struct, encoded in the expression (λ (pos) (< pos len)).

\n\n

Now we can try using a path as a sequence:

\n\n
\n \n \n \n \n \n \n \n \n \n
\n \n \n \n \n \n
> (define-cairo cairo-copy-path
    (_fun _cairo_t -> _cairo_path_t))
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
> (do-cairo (λ (ctx)
              (cairo-move-to ctx 50.0 50.0)
              (cairo-line-to ctx 206.0 206.0)
              (cairo-move-to ctx 50.0 206.0)
              (cairo-line-to ctx 115.0 115.0)
              (define path (cairo-copy-path ctx))
              ; Using path as a sequence
              (for ([elem path])
                (displayln elem))
              (cairo-stroke ctx)))
\n \n \n \n \n \n \n \n \n \n
\n

(move-to (50.0 50.0))

\n

(line-to (206.0 206.0))

\n

(move-to (50.0 206.0))

\n

(line-to (115.0 115.0))

\n

\"image\"

\n\n

Notice how the sequence prints out as an intuitive list of commands\ninstead of a bunch of opaque union values as we saw before when using\nthe _array/list type.

\n\n

That concludes part 3 of the FFI tutorial. Hopefully you’re now equipped\nto deal with union types and custom C types. If not, see the\nFFI reference\nfor more details on\nunions\nand\ncustom C types.

\n\n

Thanks to Ben Greenman for suggestions/feedback and to Sam\nTobin-Hochstadt for suggesting to cover union types!

")) ((? . 65) f post (u . "Reviews and author responses: we should stop asking for 500-word responses") (? . 65) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/08/13/reviews-and-author-responses-we-should-stop-asking-for-500-word-responses/index.html" . unix) (u . "/blog/2017/08/13/reviews-and-author-responses-we-should-stop-asking-for-500-word-responses/") (u . "2017-08-13T14:29:41") (? . 66) (? . 58) (c (u . "Author: Gabriel Scherer")) (u . "\n

This year I reviewed many ICFP submissions, and got to be on the receiving end of equally many author responses (also sometimes called, somewhat combatively, rebuttals). I found that there was a large difference between the official written advice on author responses and what I, as a reviewer reading the responses, found effective. In particular, I now believe that limiting yourself to 500 words should strongly be avoided — we should even stop giving that advice.

") #t (u . "\n

This year I reviewed many ICFP submissions, and got to be on the receiving end of equally many author responses (also sometimes called, somewhat combatively, rebuttals). I found that there was a large difference between the official written advice on author responses and what I, as a reviewer reading the responses, found effective. In particular, I now believe that limiting yourself to 500 words should strongly be avoided — we should even stop giving that advice.

\n\n\n

This year, I had the honor (and accompanying load of work) of being a Program Committee (PC) member at the ICFP conference. It was my first time being a PC member at a conference, and I found it extremely pleasant and interesting, thanks to the authors who sent us articles to review, my fellow PC members, and the ever-smiling careful balancing work of our PC chair, Mark Jones. It was also a lot of work, starting with 18 reviews to do over a month and a half, an intense PC meeting, and the new “second phase” process with the opportunity for authors and reviewers to exchange feedback on changes requested by the program committee.

\n\n

There is little guidance on how to write author responses, although this blog post by Michael Hicks on pl-enthusiast is quite good. One thing that is obvious as a reviewer and is only slightly brushed in this post, however, is that author responses should not aim to fit a 500 words limit, and in fact I believe that it is a bad idea to do so.

\n\n

As for most conference, the ICFP review system recommends (in writing) to authors to keep their response to 500 words (some systems also highlight words after those in red to make the point clear). Don’t do this! The least convincing responses I have seen are those that followed this recommendation.

\n\n

(I have also seen at least 18*2 other reviewers read the same responses I read, most of them well over 500 words, and none of them made any comment on the length of the author responses.)

\n\n

We have a frustrating situation where the explicit rule is different from the thing people do in practice. This is bad for newcomers that do not know the norms and cannot tell if ignoring the rule may hurt them. This is the point of this blog post:

\n\n\n\n

My personal response format

\n\n

My author responses start with general comments that elaborate on the main point I want to tell all reviewers. Then, a second part contains per-reviewer comments (one section per reviewer); it is clearly marked as skippable. Here is the skeleton of the last response I wrote:

\n\n
\n

We thank the reviewers for their work on our article and their detailed feedback. We start with a general discussion that responds to the salient points raised by reviewers. In a second part, we provide detailed responses to the questions/remarks of each reviewer.

\n

General discussion

\n

[..]

\n

Specific questions/comments: review #A

\n

[..]

\n

Specific questions/comments: review #B

\n

[..]

\n

Specific questions/comments: review #C

\n

[…]

\n\n

For this particular response, the “General discussion” section used 1296 words according to wc -w (M-x shell-command-on-region). In the following sections, I quote the reviews to answer specific points, email-style (following the Markdown syntax that HotCRP renders properly).

\n\n

Suggested wording for PC chairs

\n\n

If you are a PC chair, you should remove the suggestion of respecting a 500 words limit for your conference. Here would be a suggested alternative wording:

\n\n
\n

Please remember, in writing your author response, that reviewers may stop reading the response at any point. We suggest having a reasonably-sized main section where you make your most important high-level comments, and clearly marked sections where you answer individual reviewer’s questions. It is not useful nor productive to answer every point of each review, you should focus on the comments that you believe the reviewers are most interested in.

")) ((? . 67) f post (u . "Welcome to the PRL blog") (? . 67) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/04/29/welcome-to-the-prl-blog/index.html" . unix) (u . "/blog/2016/04/29/welcome-to-the-prl-blog/") (u . "2016-04-29T14:50:29") #f (? . 40) (c (u . "about") c (u . "1st blog post") c (u . "Author: Ben Greenman")) (u . "\n

Greetings, ground rules, hopes, dreams, and notes for contributors. Welcome aboard.

") #t (u . "\n

Greetings, ground rules, hopes, dreams, and notes for contributors. Welcome aboard.

\n\n\n

Earlier this year, the Programming Research Lab (PRL) was blessed with a new postdoc: Gabriel Scherer from INRIA Paris-Rocquencourt, France. Ever since Gabriel arrived things have been changing here in Boston. We now have homemade bread on the first Tuesday of every month, orange water crepes after holidays, and someone new to go out for bubble tea with in between. All that and an enthusiastic colleague and researcher.

\n\n

In his spare time between lobbying the CS department for an espresso machine and building multi-language compilers, Gabriel is also a champion of open access. Hence this blog, a window into the life and times of PRL students made possible by Gabriel’s tactical prodding and careful delegation of responsibilities. Anything you might read about in a rejected conference paper or hear over coffee is fair game here: the goal is to give the wide world a glimpse of our lab and people.

\n\n

For Contributors

\n\n

These pages are generated using Greg Hendershott’s frog static website generator. To create a new post:

\n\n
    \n
  1. Clone or fork the nuprl.github.io repository
    2. \n
  2. Check out a new git branch for your post
    3. \n
  3. Run cd blog; raco frog -n \"TITLE\" to build a template for a new post
    4. \n
  4. Add content to the new markdown file (under _src/posts)
    5. \n
  5. Rebuild the blog with raco frog -b
    6. \n
  6. Run cd ..; raco frog -p to start a web server and view your changes at http://localhost:3000/
    7. \n
  7. Send a pull request to the nuprl.github.io repo
\n\n

An open pull request is the best place to ask questions about the formatting or content of a post. We promise that within a few days of opening a PR someone with push access will reply with feedback or merge the request.

\n\n

Contributions are open to anyone: current labmates, alumni, friends from the Racket mailing list, and even recovering C programmers. One should have a strong connection to Northeastern or our research, but even that is not strictly necessary. Visitors are always welcome to the PRL.

")) ((? . 68) f post (u . "Measuring GC latencies in Haskell, OCaml, Racket") (? . 68) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/index.html" . unix) (u . "/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/") (u . "2016-05-24T10:51:34") (? . 69) (? . 1) (c (u . "garbage collection") c (u . "latency") c (u . "instrumentation") c (u . "haskell") c (u . "ghc") c (u . "ocaml") c (u . "racket") c (u . "Author: Gabriel Scherer")) (u . "\n

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

\n\n
\n

Low latency, large working set, and GHC’s garbage collector: pick two of three

\n\n

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

\n\n

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

\n\n

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

\n\n

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

\n\n

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

\n\n") #t (u . "\n

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

\n\n
\n

Low latency, large working set, and GHC’s garbage collector: pick two of three

\n\n

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

\n\n

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

\n\n

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

\n\n

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

\n\n

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

\n\n\n\n\n

The Haskell benchmark

\n\n

James Fisher’s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly — a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22
\n
\n 
import qualified Control.Exception as Exception\nimport qualified Control.Monad as Monad\nimport qualified Data.ByteString as ByteString\nimport qualified Data.Map.Strict as Map\n\ndata Msg = Msg !Int !ByteString.ByteString\n\ntype Chan = Map.Map Int ByteString.ByteString\n\nmessage :: Int -> Msg\nmessage n = Msg n (ByteString.replicate 1024 (fromIntegral n))\n\npushMsg :: Chan -> Msg -> IO Chan\npushMsg chan (Msg msgId msgContent) =\n  Exception.evaluate $\n    let inserted = Map.insert msgId msgContent chan in\n      if 200000 < Map.size inserted\n      then Map.deleteMin inserted\n      else inserted\n\nmain :: IO ()\nmain = Monad.foldM_ pushMsg Map.empty (map message [1..1000000])\n
\n
\n
\n\n

To compile and run the program (make run-haskell also works in my repository):

\n\n
ghc -O2 -optc-O3 Main.hs  # compile the program\n./Main +RTS -s            # run the program (with GC instrumentation enabled)
\n\n

On my machine, running the program takes around 1.5s. We are not interested in the total running time (the throughput of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post — and there it is considered excessive, with an expected worst-case latency of at most “a few milliseconds”.

\n\n

(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)

\n\n

This Haskell code makes two assumption about the Map data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus Map.size is constant-time — for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key — OCaml does this as well, but Racket uses hashes instead.

\n\n

I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it’s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19
\n
\n 
type Msg = ByteString.ByteString\ntype Chan = Map.Map Int Msg\n\nwindowSize = 200000\nmsgCount = 1000000\n\nmessage :: Int -> Msg\nmessage n = ByteString.replicate 1024 (fromIntegral n)\n\npushMsg :: Chan -> Int -> IO Chan\npushMsg chan highId =\n  Exception.evaluate $\n    let lowId = highId - windowSize in\n    let inserted = Map.insert highId (message highId) chan in\n    if lowId < 0 then inserted\n    else Map.delete lowId inserted\n\nmain :: IO ()\nmain = Monad.foldM_ pushMsg Map.empty [0..msgCount]\n
\n
\n
\n\n

This variant has the same running times and worst-case pause, 50ms, as the original program.

\n\n

Explaining Haskell results

\n\n

James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC’s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.

\n\n

The original blog post contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.

\n\n

As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell’s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.

\n\n

OCaml version

\n\n

The code can easily be ported into OCaml, for example as follows:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17
\n
\n 
open Batteries\nmodule IMap = Map.Make(Int)\n\nlet message n = String.make 1024 (Char.chr (n mod 256))\n\nlet window_size = 200_000\nlet msg_count = 1_000_000\n\nlet push_msg chan high_id =\n  let low_id = high_id - window_size in\n  let inserted = IMap.add high_id (message high_id) chan in\n  if low_id < 0 then inserted\n  else IMap.remove low_id inserted\n\nlet () =\n  Seq.init msg_count (fun i -> i)\n  |> Seq.fold_left push_msg IMap.empty |> ignore\n
\n
\n
\n\n

Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.

\n\n

To measure the maximal pause time, there are two options:

\n\n\n\n

To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the --with-instrumented-runtime configure-time switch. Then, you can use the i-variant (i for “instrumented”) of the runtime that is compiled with instrumentation enabled. (My makefile rule make\nrun-ocaml-instrumented does this for you, but you still need a switch compiled with the instrumented runtime.)

\n\n
ocamlbuild -tag \"runtime_variant(i)\" main.native\nOCAML_INSTR_LOG=ocaml.log ./main.native
\n\n

The log file ocaml.log will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools ocaml-instr-report and ocaml-instr-graph of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is dispatch, which contains the time spent in all GC activity. The relevant section of ocaml-instr-report’s output shows:

\n\n
==== dispatch: 2506\n470ns..1.0us:  1     (768ns)                       0.04%\n1.0us..2.2us: # 2                                  0.12%\n2.2us..4.7us: ### 8                                0.44%\n4.7us..10us : #### 10                              0.84%\n 10us..22us :  1     (14us)                        0.88%\n 22us..47us :                                      0.88%\n 47us..100us:                                      0.88%\n100us..220us: ## 3                                 1.00%\n220us..470us: ########## 668                      27.65%\n470us..1.0ms: ########### 1795                    99.28%\n1.0ms..2.2ms: ##### 17                            99.96%\n2.2ms..4.7ms:  1     (2.7ms)                     100.00%
\n\n

As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.

\n\n

The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration — as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16
\n
\n 
let worst = ref 0.\nlet time f =\n  let before = Unix.gettimeofday () in\n  let result = f () in\n  let after = Unix.gettimeofday () in\n  worst := max !worst (after -. before);\n  result\n\nlet push_msg chan high_id = time @@ fun () ->\n  let low_id = high_id - window_size in\n  let inserted = IMap.add high_id (message high_id) chan in\n  if low_id < 0 then inserted\n  else IMap.remove low_id inserted\n\n(* ..main loop.. *)\nlet () = Printf.printf \"Worst pause: %.2E\\n\" !worst\n
\n
\n
\n\n

Running this version reports a worst-case latency of 2ms seconds on my machine (I use the %E formatter for scientific notation, so it gets printed as 2.03E-03), which is in line with the instrumented runtime — actually slightly lower, as the instrumentation may add some overhead.

\n\n

A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.

\n\n

Explaining OCaml results

\n\n

The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.

\n\n

The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency instrumentation tools that I’m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.

\n\n

Racket version

\n\n

Max New wrote a first version of Racket port of this benchmark — he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19
\n
\n 
#lang racket/base\n(require racket/match)\n\n(define window-size 200000)\n(define msg-count  2000000)\n\n(define (message n) (make-bytes 1024 (modulo n 256)))\n\n(define (push-msg chan id-high)\n  (define id-low (id-high . - . window-size))\n  (define inserted (hash-set chan id-high (message id-high)))\n  (if (id-low . < . 0) inserted\n      (hash-remove inserted id-low)))\n\n(define _\n  (for/fold\n     ([chan (make-immutable-hash)])\n     ([i (in-range msg-count)])\n     (push-msg chan i)))\n
\n
\n
\n\n

I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:

\n\n\n\n

Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable PLT_INCREMENTAL_GC=1. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.

\n\n

On my machine, using the latest release Racket 6.5, the maximal pause time reported by gcstats is around 150ms, which is rather bad — the excessive pause of GHC was 50ms.

\n\n

Investigating the Racket results

\n\n

I sent an email to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.

\n\n

First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.

\n\n

Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the “ramp-up” period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran make\nrun-racket-instrumented. I can look at the pause time of major collections by doing grep MAJ racket.log, and on my machine I have:

\n\n
GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373\nGC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521\nGC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810\nGC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571\nGC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321\nGC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046\nGC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553\nGC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048\nGC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506\nGC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983\nGC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438\nGC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907\nGC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362\nGC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897\nGC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348\nGC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827\nGC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290\nGC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774\nGC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251\nGC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730\nGC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254\nGC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730\nGC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220\nGC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697
\n\n

Look at the evolution of major collection pause times: there is an early peek at 140ms, but then pause times decrease and the steady state has sensibly shorter pauses of around 22ms. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.

\n\n

My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.

\n\n

This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it’s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer’s post, and would not be a problem during the steady state of an actual message-passing application.

\n\n

Tuning the Racket version

\n\n

Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket’s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.

\n\n

I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime — around the iteration count that corresponds to the maximal channel size. I defined a function doing just that

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7
\n
\n 
(define (maybe-gc i)\n  (when (and gc-during-rampup\n             (i . > . (window-size . / . 2))\n             (i . < . (window-size . * . 2))\n             (zero? (modulo i 50)))\n        (collect-garbage 'incremental)\n        (collect-garbage 'minor)))\n
\n
\n
\n\n

which is controlled by a gc-during-rampup parameter that you can explicitly set to #t to experiment — explicit GC calls are disabled by default in my benchmark code. Then I just inserted a (maybe-gc i) call in the main loop.

\n\n

Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms — the pauses during the steady state are unchanged, around 22ms.

\n\n

This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.

\n\n

To summarize, Racket’s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.

")) ((? . 70) f post (u . "Complete Monitors for Gradual Types") (? . 70) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/10/31/complete-monitors-for-gradual-types/index.html" . unix) (u . "/blog/2019/10/31/complete-monitors-for-gradual-types/") (u . "2019-10-31T21:58:26") (? . 71) (? . 64) (c (u . "migratory typing") c (u . "gradual typing") c (u . "complete monitoring") c (u . "extended abstract") c (u . "Author: Ben Greenman")) (u . "\n

Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction — a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.

") #t (u . "\n

Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction — a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.

\n\n\n
\n

Note: this post is an extended abstract for the paper Complete Monitors for Gradual Types by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, click here.

\n\n

Example: Clickable Plot

\n\n

The program below has a subtle bug. Can you find it?

\n\n

\"Untyped

\n\n

First of all, this pseudocode program combines three chunks of code:

\n\n\n\n

The bug is in the API — in the type ([N, N]) => Image. This type promises that a given function can expect a pair of numbers, and indeed the client function h expects a pair. But the library code on the right sends a MouseEvt object.

\n\n

What happens when we run this program in a type-sound mixed-typed language? Does h receive the invalid input?

\n\n

As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.

\n\n

Type Soundness is Not Enough

\n\n

Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.

\n\n
\n

Classic Type Soundness

\n

If e : T then one of the following holds:

\n
\n\n

A mixed-typed language needs two “type soundness” theorems: one for typed code and one for untyped code. The typed soundness theorem can resemble a classic theorem. The untyped soundness theorem is necessarily a weaker statement due to the lack of types:

\n\n
\n

Mixed-Typed Soundness

\n

If e : T then one of the following holds:

\n \n

And if e is untyped then one of the following holds:

\n
\n\n

Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback h in the code above receives a pair value. We have an untyped function called from an untyped context — since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!

\n\n

\"Untyped

\n\n

Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type [N, N] to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.

\n\n

The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.

\n\n

Complete Monitoring

\n\n

A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of ownership to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.

\n\n
\n

Complete Monitoring

\n

For all well-formed e and all e', if e -->* e' then every subexpression of e' has a unique owner.

\n\n

This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.

\n\n

An Aid to Debugging

\n\n

The question raised by the Clickable Plot example is whether a language can detect one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging — either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.

\n\n

The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.

\n\n

Also in the paper, you can find:

\n\n\n\n

Paper: https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf

")) ((? . 72) f post (u . "Transient Answers Old Questions") (? . 72) 1731622765 (p+ #"/home/runner/work/website/website/blog/2020/10/15/transient-answers-old-questions/index.html" . unix) (u . "/blog/2020/10/15/transient-answers-old-questions/") (u . "2020-10-15T13:32:12") (? . 73) (? . 44) (c (u . "typed racket") c (u . "transient") c (u . "Author: Ben Greenman")) (u . "\n

Several old questions from the Typed Racket mailing list have new and simple answers under a “transient” Typed Racket.

") #t (u . "\n

Several old questions from the Typed Racket mailing list have new and simple answers under a “transient” Typed Racket.

\n\n\n
\n\n

For the past few months, I’ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the RFC and pull request.

\n\n

The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that’s about it. By contrast, types are much stronger in classic Typed Racket.

\n\n

Shallow Racket’s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the Racket mailing list. There were a surprising number of questions that went like this:

\n\n
\n

Q. Hey, I ran a program expecting X to happen, but Y happened instead. Is this a bug?

\n

A. No, Typed Racket has to do Y because of its strong types.

\n\n

… but changing to shallow types gives the X behavior! Here are their stories.

\n\n

Going forward, Deep refers to normal Typed Racket and Shallow refers to Shallow Typed Racket.

\n\n
\n\n

Higher-Order Value as Any

\n\n

Original message : groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ

\n\n

On 2018–04–16, mailoo wrote:

\n\n
\n

I play a little with the “Any” type (due to ‘dynamic-require’ which return Any), and I’m not able to cast them back in a function.

\n

I (over) simplify my question with this little program :

\n\n
(: p Any) \n(define (p i) (displayln i)) \n\n; Here I want to get back my function \n(define proc (cast p (-> Integer Void))) \n(proc 2)
\n\n
\n

but I get this error when I try to execute the function :

\n\n
; contract violation \n; Attempted to use a higher-order value passed as `Any` in untyped code: #<procedure:p>
\n\n

What’s going on?

\n\n

Deep raises an error because it must enforce the Any type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.

\n\n

How’s transient?

\n\n

Shallow prints 2 and returns void. No error. Same goes for dynamic-require.

\n\n
\n\n

Parametric Contract Affects Untyped Code

\n\n

Original message : groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ

\n\n

On 2019–12–15, John Clements wrote:

\n\n
\n

It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:

\n\n
  #lang typed/racket \n\n  (require/typed racket/list \n  [index-of (All (T) ((Listof T) T -> (U False Natural)))]) \n\n  (index-of '(n s e w) 'n) ;; returns... #f?
\n\n
\n

In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).

\n

I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.

\n\n

What’s going on?

\n\n

Deep enforces parametricity for All types, and this throws off the equality function that index-of uses internally.

\n\n

How’s transient?

\n\n

Shallow returns 0.

\n\n

ps John, thanks very much for working on Advent of Code and mailing the list!

\n\n
\n\n

Unable to Protect Opaque Value as Any

\n\n

Original message : groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ

\n\n

On 2019–12–11, Marc Kaufmann wrote:

\n\n
\n

I have one file called type-test.rkt with the following

\n\n
#lang typed/racket\n\n(require (only-in typed/web-server/http response/xexpr response))\n\n(provide f2)\n\n(: f2 (-> (U response Any)))\n(define (f2)\n  (define x '(body (h1 \"Try it\")))\n  (: resp response)\n  (define resp (response/xexpr x))\n  resp)
\n\n
\n

Then I have another untyped file for a servlet:

\n\n
#lang racket\n\n(require \"type-test.rkt\"\n         web-server/servlet\n         web-server/servlet-env)\n\n(define (start req)\n  (f2))\n\n(serve/servlet start\n               #:servlet-regexp #rx\"\"\n               #:launch-browser? #false\n               #:port 8080)
\n\n
\n

Notice that I am telling [f2] that resp is of type response. Yet, when I run the server with start [….] I get the following result:

\n

(f2): Error, see below.

\n

The error is:

\n\n
f2: broke its own contract\n  any-wrap/c: Unable to protect opaque value passed as `Any`\n  value: #<response>\n  in: the range of\n      (-> Any)
\n\n

What’s going on?

\n\n

Deep tries to enforce the Any type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.

\n\n

How’s transient?

\n\n

Shallow does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.

\n\n
\n\n

Type Inference Installs a Precise Type

\n\n

Original message : groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ

\n\n

On 2020–02–14, John Clements wrote:

\n\n
\n

I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.

\n

Here’s a program:

\n\n
#lang typed/racket \n\n(define-type Store (Mutable-HashTable Integer Value)) \n(define-type Value (U Real Boolean String)) \n\n(define top-store\n  (cast\n    (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f)))\n    Store))\n\n(hash-set! top-store 5 1234)
\n\n
\n

It fails with this error:

\n\n
contract violation\nexpected: (or/c (and/c byte? positive?) #t #f)\ngiven: 1234\nin: the values of\nthe 3rd conjunct of\n(and/c hash?\n       hash-mutable?\n       (hash/c exact-integer?\n               (or/c (and/c byte? positive?) #t #f)\n               #:immutable #f))
\n\n

What’s going on?

\n\n

First off, Deep runs fine after swapping cast for ann.

\n\n

Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.

\n\n

The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.

\n\n

How’s transient?

\n\n

Shallow runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.

\n\n
\n\n

Same-Arity Functions in a Case Lambda

\n\n

Original message : groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ

\n\n

On 2019–07–05, Ryan Kramer wrote:

\n\n
\n

In the code below, can maybe-car have the given type [….]?

\n\n
#lang typed/racket\n\n(module untyped racket\n  (provide maybe-car)\n  (define (maybe-car x)\n    (cond\n      [(pair? x) (car x)]\n      [else x])))\n\n(require/typed\n 'untyped\n [maybe-car (All (a b) (case->\n                        (-> (Pairof a b) a)\n                        (-> a a)))])
\n\n
\n

[Current error:]

\n\n
Type Checker:\n Type (All (a b) (case-> (-> (Pairof a b) a) (-> a a)))\n  could not be converted to a contract:\n   function type has two cases of arity 1
\n\n

What’s going on?

\n\n

Deep tries to enforce the type with a Racket or/c contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don’t know which to apply to an incoming function.

\n\n

Note, the “Type Checker” error message is much better than what or/c would give on its own.

\n\n

How’s transient?

\n\n

Shallow simply checks that maybe-car accepts both arities inside the case-> type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.

\n\n
\n\n

Immutable Type Affects Untyped Code

\n\n

Original message : groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ

\n\n

On 2020–02–17, Bertrand Augereau wrote:

\n\n
\n

Hello everybody, I’m trying to gradually type my script to make it a proper app (yes I’m a static-ish guy) and I have an issue (Racket 7.6 CS).

\n\n
; racket_mod.rkt:\n#lang racket\n\n(provide (struct-out s))\n(provide list-of-s)\n(provide set-list-of-s!)\n\n(struct s (a))\n(define list-of-s '())\n(define (set-list-of-s! los)\n  (set! list-of-s los))
\n\n
; racket_mod_typed.rkt:\n#lang typed/racket\n\n(provide (struct-out s2))\n(provide list-of-s2)\n(provide set-list-of-s2!)\n\n(struct s2 ([a : Natural]))\n(define list-of-s2 '())\n(define (set-list-of-s2! [los : (Listof s2)])\n  (set! list-of-s2 los))
\n\n
; racket_main.rkt:\n#lang racket\n\n(require \"racket_mod.rkt\")\n(require \"racket_mod_typed.rkt\")\n\n(define los (list (s 1) (s 2)))\n(set-list-of-s! los)\n(displayln list-of-s)\n\n(define los2 (list (s2 1) (s2 2)))\n(set-list-of-s2! los2)\n(displayln list-of-s2)
\n\n
\n

list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)

\n\n

What’s going on?

\n\n

Deep enforces the type of list-of-s2 with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.

\n\n

Here’s a step-by-step breakdown:

\n\n
    \n
  1. the typed module creates an empty list-of-s2
    2. \n
  2. the main module imports the list and receives a new copy
    3. \n
  3. the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable
    4. \n
  4. the main module reads from its copy — and it’s still empty
\n\n

How’s transient?

\n\n

Shallow lets the original list travel to untyped code. There are no contracts in the way.

\n\n

Discussion

\n\n

Wow! It’s great to see that Shallow Racket works “as expected” on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.

\n\n

If you have a similar experience with a deep-types error, let me know.

\n\n

Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error — and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.

\n\n

Shallow Typed Racket is coming soon. Follow the pull request or watch the Racket release notes for news.

\n\n

Links

\n\n\n\n

Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.

")) ((? . 26) f post (u . "Linear Types for Low-level Languages") (? . 26) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/02/28/linear-types-for-low-level-languages/index.html" . unix) (u . "/blog/2017/02/28/linear-types-for-low-level-languages/") (u . "2017-02-28T09:51:55") (? . 25) (? . 14) (c (u . "HOPL") c (u . "Author: Daniel Patterson")) (? . 5) #t (u . "\n\n

In this talk, we covered early papers (primarily, by Girard, Lafont, and Abramsky) on linear logic and its reflections into computation. The goal was to understand why linearity is often turned to as a principled way to control resource usage, as shows up in a language like Rust. From the very beginning, researchers realized the implications for “low-level” languages - that linear resources would eliminate the need for garbage collection, allow in-place mutation, and enable safe parallel computation. However, pure implementations of linearity incur lots of copying, doing away with any efficiency gained, and we covered a survey of papers that attempted to reconcile this contradiction by weakening linearity in controlled ways.

\n\n

Notes:

\n\n\n\n
\n\n

Just after the talk, over lunch, we had a lab discussion about the phrase “low level”. Here are some thoughts:

\n\n\n\n

And here are some example “low-level” tasks:

\n\n")) ((? . 62) f post (u . "Introducing HOPL 2017") (? . 62) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/02/15/introducing-hopl-2017/index.html" . unix) (u . "/blog/2017/02/15/introducing-hopl-2017/") (u . "2017-02-15T01:21:37") (? . 60) (? . 29) (c (u . "HOPL") c (u . "Author: Ben Greenman")) (u . "\n

This semester at Northeastern, Matthias Felleisen is organizing the History of Programming Languages seminar. Look for posts tagged HOPL for updates from the lectures.

") #t (u . "\n

This semester at Northeastern, Matthias Felleisen is organizing the History of Programming Languages seminar. Look for posts tagged HOPL for updates from the lectures.

\n\n\n

Once every 6 to 8 years (i.e., once every batch of Ph.D. students?), Matthias Felleisen teaches History of Programming Languages. Nominally, the course is a seminar. But unlike a typical seminar course, weekly topics are not the technical details from a handful of papers. Rather:

\n\n
\n

The primary goal is to understand (some of) the discipline as it exists today and how some of its major themes evolved.

\n\n
\n

The secondary goal is to develop basic skills for understanding and describing research themes. Every student will learn to study a theme via a series of papers, prepare an annotated bibliography, and present the key steps in the evolution of the theme.

\n\n

Themes is the operative word. To set the tone, this semester started with “themes that NUPRL faculty members have developed over the many decades of their careers.”

\n\n\n\n

At this point in the course, we are just starting with the student presentations. As these presentations happen, we plan to push updates to this blog. All presentation materials are in the course repository:

\n\n\n\n

Speakers’ notes and annotated bibliographies are in top-level folders in the repo. Discussion summaries and “unofficial” notes are in the top-level lecture_notes/ folder.

\n\n

The list of upcoming presentations is online (along with the papers each presentation is based on):

\n\n\n\n

Blogs posts for each talk should appear 2 weeks after the talk happens.

\n\n
\n\n

Links to past editions of HOPL:

\n\n")) ((? . 74) f post (u . "[Why am I going to ICFP 2017? (cross-post)](https://williamjbowman.com/blog/2017/08/29/why-am-i-going-to-icfp-2017/)") (? . 74) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/08/29/-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017/index.html" . unix) (u . "/blog/2017/08/29/-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017/") (u . "2017-08-29T13:33:21") (? . 75) (? . 18) (c (u . "Author: William J. Bowman")) (? . 5) #f (? . 5)) ((? . 76) f post (u . "On-Stack Replacement") (? . 76) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/01/28/on-stack-replacement/index.html" . unix) (u . "/blog/2019/01/28/on-stack-replacement/") (u . "2019-01-28T10:29:57") (? . 77) (? . 54) (c (u . "Author: Ming-Ho Yee")) (u . "\n

Last semester, I took a course where the final project was to write a survey paper on “a topic in the intersection between computer systems and your area.” So I wrote about on-stack replacement.

") #t (u . "\n

Last semester, I took a course where the final project was to write a survey paper on “a topic in the intersection between computer systems and your area.” So I wrote about on-stack replacement.

\n\n\n

Abstract

\n\n
\n

On-stack replacement (OSR) is a programming language implementation technique that allows a running program to switch to a different version of code. For example, a program could start executing optimized code, and then transfer to and start executing unoptimized code. This was the original use case for OSR, to facilitate debugging of optimized code.

\n

After its original use was established, OSR shifted to a different use case: optimizing programs. OSR allows the run-time system to detect if a program is executing an inefficient loop, recompile and optimize the method that contains the loop, and then transfer control to the newly compiled method. Another strategy is to optimize code based on some assumptions, then, if the assumptions are invalidated at run-time, transfer control back to the original, unoptimized code.

\n

In this survey paper, we study how OSR was first introduced as a means for debugging, how it came to be used for program optimizations, its implementation as a reusable library, and other directions of research.

\n\n

If you’re interested, you can find a copy here or on Overleaf.

\n\n
\n\n

If you liked this post, you may also be interested in tracing JITs for dynamic languages.

")) ((? . 78) f post (u . "A Spectrum of Type Soundness and Performance") (? . 78) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html" . unix) (u . "/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/") (u . "2018-10-06T11:23:35") (? . 79) (? . 34) (c (u . "migratory typing") c (u . "gradual typing") c (u . "extended abstract") c (u . "Author: Ben Greenman")) (u . "\n

The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.

") #t (u . "\n

The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.

\n\n\n
\n

Note: this post is an extended abstract for the paper A Spectrum of Type Soundness and Performance by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018

\n\n

A dynamically-typed language runs any program that “looks good” (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both “looks good” and is well-typed according to a type checker.

\n\n

A mixed-typed language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.

\n\n
\"Figure\n

Figure 1: Some mixed-typed languages

\n\n

These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:

\n\n\n\n

That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system’s protocol for checking dynamically-typed values at a type boundary (between static and dynamic code).

\n\n

Main Contribution

\n\n

In the paper A Spectrum of Type Soundness and Performance, we define a tiny mixed-typed language and show three ways to define the behavior of programs — based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.

\n\n

The three behaviors are inspired by existing languages. A higher order behavior ensures that dynamically-typed values match the static type at a boundary — by checking the value when possible, and by monitoring the value’s future interactions when necessary. A first order behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An erasure behavior does no checking whatsoever.

\n\n
\n

Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function f, then one way to enforce the type boundary is to wrap f in a proxy to assert that every future call to f returns a number. In this case, the proxy monitors the behavior of f.

\n\n

Concretely, the paper defines three formal semantics with the same names. The higher-order semantics enforces full types at the boundaries (Section 2.3). The first-order semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every “selector” operation in typed code, e.g., function application, data structure access (Section 2.5). The erasure semantics simply ignores the types (Section 2.4).

\n\n

Each semantics satisfies a different notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the online supplement gives full proofs.

\n\n

The paper has more to say about the meta-theory. See section 2 and section 4.

\n\n
\n

To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for subset of the language (in the spirit of soundiness) or use the name “type soundness” to describe a different property.

\n\n

Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations — similar to Typed Racket’s no-check language.

\n\n

Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (POPL 2016). The programs we measured are the non-object-oriented ones from our benchmark suite.

\n\n

To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises — see section 3 of the paper, section B of the supplement, and/or the slides.

\n\n

Implications

\n\n
    \n
  1. The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the supplement, or slide 114.
    2. \n
  2. Programmers using mixed-typed languages need to know what guarantees their types provide. (It is not safe to assume that TypeScript types give the same guarantees as OCaml types!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.
    3. \n
  3. The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?
\n\n
\n

Note: in this post, a mixed-typed language is one that allows any combination of static and dynamic typing. A gradually-typed language is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (SNAPL 2015).

")) ((? . 79) f post (u . "Sampling Gradual Typing Performance") (? . 79) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/05/08/sampling-gradual-typing-performance/index.html" . unix) (u . "/blog/2018/05/08/sampling-gradual-typing-performance/") (u . "2018-05-08T15:37:37") (? . 59) (? . 78) (c (u . "gradual typing") c (u . "migratory typing") c (u . "performance") c (u . "statistics") c (u . "Takikawa constant") c (u . "Author: Ben Greenman") c (u . "Author: Zeina Migeed")) (u . "\n

This post explains the sampling method introduced in the paper On the Cost of Type-Tag Soundness

") #t (u . "\n

This post explains the sampling method introduced in the paper On the Cost of Type-Tag Soundness

\n\n\n

Quick Reference: How to apply the method

\n\n
    \n
  1. Find an untyped program, measure its running time.
    2. \n
  2. Define a granularity for type annotations (by-function, by-module, by-program, ….).
    3. \n
  3. Define a sample size s and number of samples r.
    4. \n
  4. Randomly select s configurations uniformly at random, measure their running time.
    5. \n
  5. Repeat the previous step r times.
    6. \n
  6. Pick a positive real number D.
    7. \n
  7. Count the proportion of configurations in each sample with running time less-than-or-equal-to D
    8. \n
  8. Build a 95% confidence interval for the r proportions computed in the previous step
    9. \n
  9. Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to D
\n\n

Background: what to measure

\n\n

A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for “adding static typing” has a few steps:

\n\n\n\n

If the semantics for statically-typed parts of the program is not the same as the semantics for dynamically-typed parts, then it is important to measure performance.

\n\n

The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.

\n\n

An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?

\n\n

The next two sections give two methods for answering this question.

\n\n

Exhaustive Method

\n\n

One way to answer our easier question is to remove type annotations one “unit” at a time and measure the running time of all these partially-typed programs. We call the “unit” the granularity of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the “partially-typed programs” the configurations of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity — I can’t just use the word configurations without telling you the granularity I have in mind.

\n\n

After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number D and count the number of configurations that run at most D times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if you are willing to accept at most a Dx slowdown, and you add your own type annotations to your own program, then there’s some hope that your configuration runs at most D times slower than your original program.

\n\n
\n

Credit for the exhaustive method: Is Sound Gradual Typing Dead? and Toward Practical Gradual Typing

\n\n

Simple Random Approximation Method

\n\n

The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.

\n\n

On the Cost of Type-Tag Soundness, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most D times slower than the original dynamically-typed program, we:

\n\n\n\n

The method is outlined above, described in the paper, and validated in that paper’s appendix. Please let us know if you have more questions.

\n\n
\n

Maybe you’re wondering, “gee why do they keep writing out ‘configurations that run at most ….’ instead of something shorter?”. Well, the short version is D-deliverable and it was introduced in the Is Sound Gradual Typing Dead? paper. Unfortunately, (1) that paper instantiated D to 3-deliverable in order to explain a few graphs and (2) at least two published papers (paper 1, paper 2) now cite us as saying 3x overhead is the cutoff between a good migratory typing system and a bad one.

\n

…

\n

If we can’t trust scientists to understand, then we definitely can’t trust you folks on the internet.

\n\n

FAQ

\n\n

Q. What is the sampling method useful for?

\n\n\n\n

Q. What is the sampling method not useful for?

\n\n\n\n

Q. Why is it okay to choose D after collecting the samples?

\n\n

The “quick reference” at the top of this post suggests choosing a value for D (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of D affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn’t and value that affects the bottom line be fixed before taking samples? (To avoid accidental data dredging.)

\n\n

The reason it is ok to pick D after taking the sample is that the running times in the sample are independent of the choice of D.

\n\n

For example, if one person chose D=3 and a second person chose D=9, both would follow the same protocol independent of D to take samples.

\n\n

Q. How does migratory typing relate to gradual typing?

\n\n

Gradual typing is not just about adding a type system to an existing programming language. See Refined Criteria for Gradual Typing and Migratory Typing: 10 Years Later for details.

\n\n

Q. Do you have code I can use to plot sampling data?

\n\n

Yes, start here:

\n\n\n\n

Please ask questions and open issues if you have trouble. The source is here:

\n\n\n\n

Q. Where is code for the sampling paper?

\n\n

Start here:

\n\n\n\n

Source is here:

\n\n\n\n

Closing Thoughts

\n\n

Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.

")) ((? . 61) f post (u . "Fall 2016 PL Junior Retrospective") (? . 61) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/01/02/fall-2016-pl-junior-retrospective/index.html" . unix) (u . "/blog/2017/01/02/fall-2016-pl-junior-retrospective/") (u . "2017-01-02T16:39:37") (? . 43) (? . 60) (c (u . "PL Junior") c (u . "Author: Ben Chung") c (u . "Author: Milo Davis") c (u . "Author: Ming-Ho Yee") c (u . "Author: Sam Caldwell")) (u . "\n

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

\n\n
    \n
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. \n
  2. they represent a significant increase in the reasoning power of types over programs.
") #t (u . "\n

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

\n\n
    \n
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. \n
  2. they represent a significant increase in the reasoning power of types over programs.
\n\n\n

There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by posts like these. So we followed the pi-for-all lecture series given by Stephanie Weirich at OPLSS, which focuses on implementing a simple dependently-typed programming language.

\n\n

After the pi-for-all lectures, we read chapter two of Edwin Brady’s dissertation on implementing dependently typed languages. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.

\n\n

Along the way, we became sidetracked by Girard’s paradox. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.

\n\n

It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a paper that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.

\n\n

The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.

\n\n

With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.

\n\n

This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.

\n\n

Thoughts on the format

\n\n

Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).

\n\n

[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]

\n\n

If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the ATTAPL chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there

\n\n

For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.

")) ((? . 49) f post (u . "[Bridging the System Configuration Gap (Cross-Post)](https://aaronweiss.us/posts/2017-06-05-bridging-the-system-configuration-gap.html)") (? . 49) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/06/09/-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html/index.html" . unix) (u . "/blog/2017/06/09/-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html/") (u . "2017-06-09T13:36:56") (? . 48) (? . 8) (c (u . "Author: Aaron Weiss")) (? . 5) #f (? . 5)) ((? . 75) f post (u . "Closure Conversion as CoYoneda") (? . 75) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/08/28/closure-conversion-as-coyoneda/index.html" . unix) (u . "/blog/2017/08/28/closure-conversion-as-coyoneda/") (u . "2017-08-28T10:30:00") (? . 58) (? . 74) (c (u . "Yoneda") c (u . "coYoneda") c (u . "category theory") c (u . "compilers") c (u . "closure conversion") c (u . "math") c (u . "Author: Max New")) (u . "\n

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

\n\n

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

\n\n

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

\n\n

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

") #t (u . "\n

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

\n\n

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

\n\n

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

\n\n

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

\n\n\n

\\(\n\\newcommand{\\Set}{\\mathsf{Set}}\n\\newcommand{\\Hom}{\\mathsf{Hom}}\n\\)

\n\n

Polymorphic Closure Conversion

\n\n

Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions’ free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.

\n\n

For instance

\n\n
let x = 3 in λ y. x + y
\n\n

would be converted to something like

\n\n
let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)
\n\n

Can we give a type to the resulting code? The source program has type Number -> Number, but the target has a type more like

\n\n
{ x: Number} × ({x : Number} × Number -> Number).
\n\n

In addition to being ugly, this type is leaking irrelevant details of the function’s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \\(\\beta\\)-reducing the term to just λ y. 3 + y would not even preserve typing. Not only that, but some bad code could now supply a different, well-typed value for x than allowed which could break invariants the programmer had about the function.

\n\n

We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is irrelevant to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be abstract to the caller and concrete to the callee. Existential types capture this.

\n\n

We can translate functions in the source of type A -> B to pairs of an environment and a code pointer, but now making the environment type existentially quantified:

\n\n
∃ Γ. Γ × (Γ × A -> B).
\n\n

Then the syntax of existential types ensure that all any consumer can do with the env : Γ in the pair is pass it to the code pointer with an A argument.

\n\n

How do we prove that this is correct? And what does correct even mean? We’ll focus on a property called full abstraction which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \\(\\beta,\\eta \\) equivalence, so things like as above:

\n\n
let x = 3 in λ y. x + y\n≡\nλ y. 3 + y
\n\n

To prove this we’ll show that in a language with existential types the types ∃ Γ. Γ × (Γ × A -> B) and A \\to B are isomorphic. The usual proof is by parametricity, instead we’ll use a closely related category-theoretic argument: the coYoneda lemma.

\n\n

The CoYoneda Lemma

\n\n

The coYoneda lemma is a generalization of the equivalence described above. I’ll start with the ordinary version which uses coends and presheaves.

\n\n

The coYoneda lemma says that for any category \\( C \\), presheaf \\( Q : C^{op} \\to \\Set \\), and object \\(A \\in C \\), \\(Q(A) \\) is isomorphic to the coend: \\[ \\exists B. (A \\to B) \\times Q(B) \\] Let’s break that down.

\n\n

Coends

\n\n

A coend is a construction that is very similar to the parametric existential quantifier. If you’re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.

\n\n

You can take the coend of a functor of type \\(M : C^{op} \\times C \\to\n\\Set \\). We can get such an \\(M \\) from a type with a free type variable like \\( X \\times A \\to X \\) by splitting the \\(X \\) into positive and negative occurrences: \\(X^- \\times A \\to X^+ \\). Then the coend \\(\\exists X. M(X,X) \\in \\Set \\) is like the union of all \\(M(X,X) \\), but where the \\(X \\) is ensured to be “irrelevant”.

\n\n

So for any object \\(A \\in C \\) there is a map \\(pack_A : M(A,A) \\to\n\\exists X. M(X,X) \\), we can “hide the A”. To make sure the \\(X \\) is treated opaquely, we add an invariance condition that says if you have an \\(mA : M(A,A) \\) and an \\(mB :\nM(B,B) \\) such that the \\(A, B\\) positions are related by some function \\(f : A \\to B \\), then \\(pack_A(mA) = pack_B(mB)\\). More formally, this means that if you have a \\(m' : M(B,A) \\), then

\n\n

\\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\\] or in a point-free style: \\[ pack_B \\circ M(B,f) = pack_A \\circ M(f,A) : M(B,A) \\to \\exists X. M(X,X) \\]

\n\n

A function parameterized by types like \\(pack \\) that has this property is called a co-wedge from \\(M \\).

\n\n

A coend is an object \\(\\exists X. M(X,X) \\) and a co-wedge \\(\\forall\nA. pack_A : M(A,A) \\to \\exists X. M(X,X) \\) that are universal, i.e. any other co-wedge \\(\\forall A. f_A : M(A,A) \\to C\\) factors through \\(pack_A \\). This gives us the syntax for existential elimination.

\n\n

If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all relations implies that a parametric \\(pack, \\exists X. M(X,X) \\) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don’t act like functions.

\n\n

Presheaves

\n\n

Next, a presheaf is just a functor \\( Q : C^{op} \\to\n\\Set \\). Think of this as a set that is parameterised by a type of “inputs”, so if you have a map in \\(C, f : A \\to B\\) you get a function \\(Q(f) :\nQ(B) \\to Q(A) \\) that “preprocesses” the inputs using \\(f\n\\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.

\n\n

So the informal explanation of the coYoneda lemma is that for any presheaf \\(Q \\), if we have an \\( \\exists X. (A \\to X) \\times Q(X)\n\\), then since we can’t inspect the \\(X \\) in any way, all we can really do is compose the \\(Q(X) \\) with the preprocesser from the function \\(A \\to X \\), giving us a \\(Q(A) \\).

\n\n

Enriched Categories and Enriched CoYoneda

\n\n

But there’s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \\(Q(A) \\) and \\(\\exists B. (A \\to B) \\times Q(B) \\) are isomorphic as sets, but we wanted an isomorphism of types in our programming language. We can reconcile this by considering enriched category theory and the enriched coYoneda lemma. Let \\(V \\) be a category, then if \\( V\n\\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word “set” with “object of \\(V \\)”.

\n\n

Specifically, a \\(V \\)-enriched category (or just \\(V\\)-category) has a set of objects \\(Ob \\), but for each pair of objects \\(A,B\n\\in Ob \\) we get a \\( V\\)-object \\(\\Hom(A,B) \\) of morphisms from \\(A \\) to \\( B \\). If \\(V \\) is a closed category, we can see \\(V \\) itself as a \\(V\\)-enriched category with the same objects and just making \\(\\Hom(A,B) = A \\to B \\) i.e. the internal hom aka exponential.

\n\n

Then we can reinterpret the coYoneda lemma above by saying \\(C \\) is a \\(V\\)-category and \\(Q \\) is a \\(V\\)-presheaf i.e., just a contravariant functor from \\(V \\) to itself: \\(Q : V^{op} \\to V\\) where the preprocessing function is now a morphism in \\(C \\). Haskelletons just call this a contravariant functor. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.

\n\n

Now that we’re working all inside our language, let’s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions

\n\n
f : (Contravariant Q) => (∃ Γ. (Δ -> Γ) × (Q Γ)) -> Q Δ\ng : (Contravariant Q) => Q Δ -> ∃ Γ. (Δ -> Γ) × (Q Γ)
\n\n

If you try to implement them you won’t be able to get it wrong, but here they are:

\n\n
f (k, qΓ) = contramap k qΓ\ng qΔ = (id, qΔ)
\n\n

where we just instantiate \\(\\Gamma = \\Delta \\) in the second case. You can prove \\( f \\circ g = id \\) using just \\(\\beta \\) and the Contravariant laws, but to prove \\(g \\circ f = id \\) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.

\n\n

Closure Version as CoYoneda

\n\n

Now it’s time to bring it all together. Let \\(V \\) be our programming language viewed as a category in the usual way.

\n\n

We want to prove the closure conversion isomorphism:

\n\n

\\[ A \\to B \\cong \\exists \\Gamma. \\Gamma \\times (\\Gamma \\times A \\to B)\n\\]

\n\n

using the \\(V \\)-coYoneda lemma which says for any contravariant functor \\(Q : V^{op} \\to V \\), and object \\(\\Delta \\in V\\)

\n\n

\\[ Q(\\Delta) \\cong \\exists \\Gamma. (\\Delta \\to \\Gamma) \\times Q(\\Gamma)\n\\]

\n\n

Clearly based on the right hand side, \\(Q \\) should be \\( - \\times\nA \\to B \\) which gives us for any \\(\\Delta \\in V\\):

\n\n

\\[ \\Delta \\times A \\to B \\cong \\exists \\Gamma. (\\Delta \\to \\Gamma) \\times (\\Gamma \\times A \\to B)\n\\]

\n\n

Next we pick \\(\\Delta = 1\\), the unit type. Then we use some basic facts about the unit type: \\(1 \\times A \\cong\nA \\) and \\(1 \\to \\Gamma \\cong \\Gamma\\) (at least in a pure language) to get the desired result by composition:

\n\n

\\[ A \\to B \\cong 1 \\times A \\to B \\cong \\exists \\Gamma. (1 \\to\n\\Gamma) \\times (\\Gamma \\times A \\to B) \\cong \\exists \\Gamma. \\Gamma\n\\times (\\Gamma \\times A \\to B)\\]

\n\n

Conclusion

\n\n

Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in Haskell and is also central to the Day Convolution, which can be used to give semantics to separation logic.

\n\n

Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we’re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.

\n\n

You might also like

\n\n")) ((? . 80) f post (u . "Racket 6.9 and Windows 10 Creators Update") (? . 80) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/index.html" . unix) (u . "/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/") (u . "2017-05-26T17:00:28") (? . 50) (? . 81) (c (u . "racket") c (u . "windows") c (u . "bugs") c (u . "Author: Leif Andersen")) (u . "\n

Racket 6.9 was released in April and it has been smooth sailing for many people. However, some people using the Windows 10 Creators Update have been experiencing crashes, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.

\n\n

The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.

\n\n

A VirtualBox image with Racket preinstalled can be downloaded here:

\n\n\n\n

The username and password for this machine are both racket.

") #t (u . "\n

Racket 6.9 was released in April and it has been smooth sailing for many people. However, some people using the Windows 10 Creators Update have been experiencing crashes, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.

\n\n

The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.

\n\n

A VirtualBox image with Racket preinstalled can be downloaded here:

\n\n\n\n

The username and password for this machine are both racket.

\n\n\n
    \n
  1. \n

    The first thing you need to install is virtualization software. In principle it doesn’t matter what you install, but for this tutorial, we will use VirtualBox. Go to their downloads page and download and install the version for your platform (most likely Windows).

    2. \n
  2. \n

    Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which you can download here. The rest of this tutorial will assume you are using this image.

    3. \n
  3. \n

    Start up VirtualBox and import the virtual machine. You can do this by clicking on File -> Import Appliance. In the dialog, select the image you downloaded and hit Continue. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click Import.

    4. \n
  4. \n

    After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit Start. Once started up, you will see DrRacket and Firefox on the VM’s desktop.

")) ((? . 82) f post (u . "Tracing JITs for Dynamic Languages") (? . 82) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/03/15/tracing-jits-for-dynamic-languages/index.html" . unix) (u . "/blog/2017/03/15/tracing-jits-for-dynamic-languages/") (u . "2017-03-15T10:54:39") (? . 83) (? . 21) (c (u . "HOPL") c (u . "Author: Ming-Ho Yee")) (? . 5) #t (u . "\n\n

Traditional JIT (just-in-time) compilers are method-based: they compile “hot” (i.e. frequently executed) methods to native code. An alternative is trace-based or tracing JITs, where the compilation unit is a (hot) sequence of instructions. Typically, such sequences of instructions correspond to loops, where programs spend most of their execution time.

\n\n

Where did the idea of tracing come from? What was appealing about it? How was tracing adapted for JITs and dynamic languages? What happened to Mozilla’s TraceMonkey, which used to be part of Firefox? Do any JITs today use tracing?

\n\n

In this talk, I trace tracing JITs from their origins to some of their recent developments. I cover five papers: the original tracing paper, an implementation of a tracing JIT for Java, the TraceMonkey JIT for JavaScript, PyPy’s “meta-level” tracing, and a specific class of optimizations for tracing JITs.

\n\n

(The idea of using the phrase “trace tracing JITs” is from Matthias Felleisen.)

\n\n

All materials can be found in the course repository:

\n\n\n\n
\n\n

If you liked this post, you may also be interested in on-stack replacement.

")) ((? . 71) f post (u . "Four Kinds of Scoping in R") (? . 71) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/09/10/four-kinds-of-scoping-in-r/index.html" . unix) (u . "/blog/2019/09/10/four-kinds-of-scoping-in-r/") (u . "2019-09-10T11:00:00") (? . 38) (? . 70) (c (u . "scope") c (u . "r") c (u . "Author: Ming-Ho Yee")) (u . "\n

In the first and second parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.

\n\n

In this third and final part of my blog series, I’d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.

\n\n

This is a “bonus” blog post, because I’m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.

") #t (u . "\n

In the first and second parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.

\n\n

In this third and final part of my blog series, I’d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.

\n\n

This is a “bonus” blog post, because I’m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.

\n\n\n

Lexical scope and statistical computation

\n\n

In Lexical Scope and Statistical Computation,1 Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.

\n\n

For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:

\n\n\n\n

Note that under this set of definitions, static scoping is a separate scoping rule and not another name for lexical scoping.

\n\n

It is possible to simulate each of strategies in R. For fun, we can even construct “factories” that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)

\n\n

Template

\n\n

Our examples will follow the template given below:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15
\n
\n 
factory <- function(fun) {\n  <???>\n}\n\nx <- 0\nh <- function() {\n  x <- 1\n  factory(function(a) x+a)\n}\ng <- h()\nf <- function() {\n  x <- 2\n  g(0)\n}\nf() # error, 0, 1, or 2\n
\n
\n
\n\n

We want to define a factory that takes a function literal and returns a closure that implements the desired scoping rule.

\n\n

Our example consists of three definitions of x. On line 5, we assign 0 to x at the top level. On line 7, we assign 1 to x inside function h, where we also create the closure. On line 12, we assign 2 to x inside the function f and right before we call g, which is the closure.

\n\n

Finally, we call f and observe the result:

\n\n\n\n

We will implement the body of factory in only 3–5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to factory on line 10.

\n\n

Trivial scoping

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17
\n
\n 
makeTrivial <- function(fun) {\n  res <- eval(substitute(fun))\n  environment(res) <- baseenv()\n  res\n}\n\nx <- 0\nh <- function() {\n  x <- 1\n  makeTrivial(function(a) x+a)\n}\ng <- h()\nf <- function() {\n  x <- 2\n  g(0)\n}\nf() # Error in f(0) : object 'x' not found\n
\n
\n
\n\n

substitute returns the unevaluated parse tree for fun. In other words, it obtains the literal argument that was passed for fun. This works because of call-by-need semantics in R: function arguments are packaged up into promises. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek2 discusses laziness in R in more detail.

\n\n

In this example, on line 8, we call factory with function(a) x+a as the argument for the formal parameter fun. Then, we evaluate that parse tree with eval.

\n\n

At this point, res is the closure with expression function(a) x+a and a reference to the environment of makeTrivial. On line 3, we change that reference to baseenv(), which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.

\n\n

Therefore, variable lookup in the function literal will only search the base environment, so f() results in an error.

\n\n

Static scoping

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17
\n
\n 
makeStatic <- function(fun) {\n  res <- eval(substitute(fun))\n  environment(res) <- globalenv()\n  res\n}\n\nx <- 0\nh <- function() {\n  x <- 1\n  makeStatic(function(a) x+a)\n}\ng <- h()\nf <- function() {\n  x <- 2\n  g(0)\n}\nf() # 0\n
\n
\n
\n\n

For this example, on line 3, we update the environment of res to refer to globalenv(), which is the top-level environment where globals are defined.

\n\n

Therefore, variable lookup searches the top-level environment, so f() returns 0.

\n\n

Lexical scoping

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17
\n
\n 
makeLexical <- function(fun) {\n  res <- eval(substitute(fun))\n  environment(res) <- parent.frame()\n  res\n}\n\nx <- 0\nh <- function() {\n  x <- 1\n  makeLexical(function(a) x+a)\n}\ng <- h()\nf <- function() {\n  x <- 2\n  g(0)\n}\nf() # 1\n
\n
\n
\n\n

Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of res to parent.frame(), which is the environment of the function (h) that called the current function (makeLexical). This allows us to simulate lexical scoping, as if the function literal was evaluated inside h, rather than makeLexical.

\n\n

Therefore, variable lookup searches the environment of h, so f() returns 1.

\n\n

Dynamic scoping

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19
\n
\n 
makeDynamic <- function(fun) {\n  function(...) {\n    res <- eval(substitute(fun))\n    environment(res) <- parent.frame()\n    res(...)\n  }\n}\n\nx <- 0\nh <- function() {\n  x <- 1\n  makeDynamic(function(a) x+a)\n}\ng <- h()\nf <- function() {\n  x <- 2\n  g(0)\n}\nf() # 2\n
\n
\n
\n\n

For this example, we need another level of indirection. makeDynamic returns an anonymous function literal. The anonymous function takes ..., which represents an arbitrary list of arguments, and then on line 5 we call res with those exact arguments. Note that we set the environment of res to be the environment of the caller of the anonymous function. Because of the multiple levels of indirection, the caller is f, on line 17.

\n\n

On line 12, makeDynamic returns a closure for the anonymous function. h returns that closure when it is called, and assigns it to g. When g is called on line 17, the function literal function(a) x+a is finally evaluated, and its environment is set to the environment of f, the caller of g.

\n\n

Therefore, variable lookup searches the environment of f, so f() returns 2.

\n\n

Conclusion

\n\n

Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the previous post, it’s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.

\n\n

And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.

\n\n

I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.

\n\n
\n\n

References

\n\n
\n
    \n
  1. \n

    R. Gentleman and R. Ihaka. \"Lexical Scope and Statistical Computing, Journal of Computational and Graphical Statistics, vol. 9, no. 3, 2000. [DOI][Available online] ↩

    2. \n
  2. \n

    A. Goel and J. Vitek. “On the Design, Implementation and Use of Laziness in R,” in Proceedings of the ACM in Programming Languages (PACMPL), vol. 3, no. OOPSLA, 2019. To appear. [Available online] ↩

")) ((? . 69) f post (u . "Gradual Typing Across the Spectrum") (? . 69) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/05/18/gradual-typing-across-the-spectrum/index.html" . unix) (u . "/blog/2016/05/18/gradual-typing-across-the-spectrum/") (u . "2016-05-18T07:58:56") (? . 40) (? . 68) (c (u . "gradual typing") c (u . "PI meeting") c (u . "Author: Asumu Takikawa")) (u . "\n
\n

Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. — Matthias Felleisen

\n\n

Yesterday we hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found here). In case you aren’t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.

\n\n

Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing—the ideas that cross language and platform barriers.

") #t (u . "\n
\n

Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. — Matthias Felleisen

\n\n

Yesterday we hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found here). In case you aren’t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.

\n\n

Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing—the ideas that cross language and platform barriers.

\n\n\n

There were a good number of both institutions and programming languages represented at the meeting, with researchers from all of Brown University, Indiana University, Northeastern University, and the University of Maryland. The languages that we work on cover a broad subset of the dynamically-typed languages: Clojure, JavaScript, R, Racket, Ruby, Pyret, and Python.

\n\n
\"\"\n

\n\n

The specific research artifacts that were represented include Reticulated Python, RDL (contracts for Ruby), StrongScript, Typed Clojure, and Typed Racket.

\n\n

In this blog post, I’ll summarize some of the key research themes that were brought up at the meeting. Since I can’t go into too much detail about every topic, I will link to the relevant research papers and other resources.

\n\n

At a high level, the talks covered four major facets of gradual typing: expressiveness, performance, usability, and implementation techniques.

\n\n

Expressiveness

\n\n

By expressiveness, I mean what kinds of language features a gradual type system supports and the richness of the reasoning that the type system provides. Since gradual typing is about augmenting existing dynamically-typed languages, a gradual type system should support the language features that programmers actually use.

\n\n

This is why recent implementations of gradual typing have focused on enabling object-oriented programming, since objects are widely used in nearly all dynamically-typed languages in use today. Unfortunately, since different languages have wildly different object systems, it’s hard to compare research on gradual OO languages. Ben Chung is working to address this by coming up with a formal model that tries to unify various accounts of objects in order to better explain the design tradeoffs. His goal is to cover the core ideas in Reticulated Python, StrongScript, and Typed Racket.

\n\n

Of course, dynamically-typed languages have a lot more than OO features. Along these lines, I gave a talk on how at NU we’re working to extend Typed Racket to cover everything from objects (my thesis topic), first-class modules (Dan Feltey’s MS project), and higher-order contracts (Brian LaChance’s MS project).

\n\n

On the other side, as programs get more complex, programmers may wish to write richer type specifications that provide even more guarantees. This makes gradual typing a wide spectrum that goes from completely untyped, fully typed, and then beyond to dependently typed. Andrew Kent and David Christiansen both presented work that takes gradual typing beyond ordinary typed reasoning with dependent types.

\n\n

Andrew presented an extension of Typed Racket that adds type refinements that can check rich properties (like vector bounds) that are found in real Racket code (see his RacketCon talk and recent PLDI paper). David Christiansen followed with a talk about adding dependent type theory to Typed Racket, which would allow correct-by-construction programming using a Nuprl-like proof system (he had a very cool GUI proof assistant demo in his slides!).

\n\n

Performance

\n\n
\"\"\n

\n\n

One of the key practical concerns about gradual typing is its performance overhead. It’s a concern because in order to ensure type safety, a gradually-typed language implementation needs to install dynamic checks between the typed and untyped parts of a program. This catches any inconsistencies between the typed interfaces and how the untyped code may call into them.

\n\n

Ben Greenman gave an upbeat talk that set the stage for this topic, pointing out some key lessons that we’ve learned about performance from building Typed Racket. The main idea he presented (also the topic of our POPL 2016 paper) is that to evaluate a gradual type system, you want to explore different ways of adding types to a program and see how much it costs. This evaluation effort started with Typed Racket, but he and Zeina Migeed are working on expanding it to Reticulated Python.

\n\n

From IU, Andre Kuhlenschmidt and Deyaaeldeen Almahallawi are exploring how ahead-of-time (AOT) compilation strategies could help reduce the cost of gradual typing. In particular, they are working on implementing Schml: a compiler from the gradually-typed lambda calculus to C.

\n\n

In addition to AOT compilation, the folks at IU are exploring tracing JIT compilation as a means to make gradual typing faster. More specifically, Spenser Bauman talked about Pycket, an alternative implementation of Racket that uses RPython/PyPy to dramatically lower the overhead of gradual typing (also see the recording of Spenser’s talk on the topic at RacketCon and his ICFP paper).

\n\n

Usability

\n\n

On the usability side, both Shriram Krishnamurthi and Ambrose Bonnaire-Sergeant made observations on what it takes to get gradual typing in the hands of real software developers.

\n\n

Shriram approached the topic from the angle of CS education, which is the focus of the Pyret language, and shared what the Brown language group is working on. While Pyret doesn’t exactly fit the mold of gradual typing, it’s a close cousin since it’s a dynamically-typed language that explicitly takes design cues from the best parts of statically-typed languages. That approach lets CS beginners think in terms of types (the approach spearheaded by HtDP and Bootstrap) without having to battle a typechecker from the start.

\n\n

For professional software developers, a major concern with gradual typing is that writing type annotations may be a tedious and time intensive task. Ambrose, who is the creator of Typed Clojure, shared some preliminary work on how to cut down on the tedium by inferring gradual type annotations by instrumenting programs for a dynamic analysis. The hope is to be able to infer both recursive and polymorphic type annotations automatically from tests (you may also be interested in Ambrose’s recent ESOP paper on Typed Clojure).

\n\n

Implementation Techniques

\n\n

Finally, several talks focused on alternative implementation techniques for gradual typing that provide a variety of software engineering benefits for implementers.

\n\n

From Maryland, Brianna Ren gave a talk on Hummingbird, a just-in-time typechecker for Ruby programs (also see the upcoming PLDI paper by Brianna and Jeff Foster). The basic idea is that it’s hard to implement a traditional static typechecker for a language that heavily relies on metaprogramming, in which the fields/methods of classes may be rewritten at run-time. This is particularly tricky for frameworks like Ruby on Rails. Instead of checking types at compile-time, Hummingbird actually executes the typechecker at run-time in order to be able to accurately check programs that use run-time metaprogramming. To reduce overheads, she uses a cache for typechecking that is invalidated when classes are modified.

\n\n

Stephen Chang gave a very different view on metaprogramming in his talk, which focused on implementing typecheckers using metaprogramming (the trivial Typed Racket library is an offshoot of this work). His key idea is that typecheckers share many aspects with macro-based metaprogramming systems, such as the need to traverse syntax and annotate it with information. Since they share so much in common, why not just implement the typechecker as a macro? Stephen demonstrates that not only is this possible, but that it’s possible to implement a wide variety of type system features this way including (local) type inference. The connection to gradual typing is that even a gradual type system can be implemented as a metaprogram by integrating the generation of dynamic checks into the macro transformation process.

\n\n

The last talk of the day (but certainly not the least), was by Michael Vitousek, who focused on the transient implementation of gradual typing (first described in his DLS paper). Traditionally, gradual type systems have implemented their dynamic checks using proxy objects that wrap method implementations with both pre- and post-checks. Unfortunately, this implementation technique often conflicts with the underlying language. Since proxying changes the identity of an object it can interfere with object equality tests. Instead, the transient approach bakes the dynamic checks into and throughout the typed code to implement a “defense in depth” against inconsistencies with untyped code. The great thing about this implementation technique is that it doesn’t demand any specialized support from the underlying language runtime and is therefore easy to port to other languages (like JavaScript).

\n\n

Conclusion

\n\n
\"\"\n

\n\n

Hopefully this blog post helps provide a better picture of the state of gradual typing research. The exciting thing about gradual typing is that it contains both interesting theoretical problems and also connects to the practical needs of software developers.

")) ((? . 81) f post (u . "Syntactic parametricity strikes again") (? . 81) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/06/05/syntactic-parametricity-strikes-again/index.html" . unix) (u . "/blog/2017/06/05/syntactic-parametricity-strikes-again/") (u . "2017-06-05T14:27:44") (? . 80) (? . 48) (c (u . "Author: Gabriel Scherer") c (u . "Author: Li-Yao Xia")) (u . "\n

In this blog post, reporting on a collaboration with Li-Yao Xia, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely “syntactic” way, by looking at the structure of canonical derivations. More precisely, I prove that \\(\n\\newcommand{\\List}[1]{\\mathsf{List}~#1}\n\\newcommand{\\Fin}[1]{\\mathsf{Fin}~#1}\n\\newcommand{\\Nat}[1]{\\mathbb{N}}\n\\newcommand{\\rule}[2]{\\frac{\\displaystyle \\array{#1}}{\\displaystyle #2}}\n\\newcommand{\\judge}[2]{{#1} \\vdash {#2}}\n\\newcommand{\\emptyrule}[1]{\\begin{array}{c}\\\\[-1em] #1 \\end{array}}\n ∀α. \\List α → \\List \\alpha\n\\) is isomorphic to \\(\n Π(n:\\Nat{}). \\List{(\\Fin{n})}\n\\) where \\(\\Fin{n}\\) is the type of integers smaller than \\(n\\), corresponding to the set \\(\\{0, 1, \\dots, n-1\\}\\).

") #t (u . "\n

In this blog post, reporting on a collaboration with Li-Yao Xia, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely “syntactic” way, by looking at the structure of canonical derivations. More precisely, I prove that \\(\n\\newcommand{\\List}[1]{\\mathsf{List}~#1}\n\\newcommand{\\Fin}[1]{\\mathsf{Fin}~#1}\n\\newcommand{\\Nat}[1]{\\mathbb{N}}\n\\newcommand{\\rule}[2]{\\frac{\\displaystyle \\array{#1}}{\\displaystyle #2}}\n\\newcommand{\\judge}[2]{{#1} \\vdash {#2}}\n\\newcommand{\\emptyrule}[1]{\\begin{array}{c}\\\\[-1em] #1 \\end{array}}\n ∀α. \\List α → \\List \\alpha\n\\) is isomorphic to \\(\n Π(n:\\Nat{}). \\List{(\\Fin{n})}\n\\) where \\(\\Fin{n}\\) is the type of integers smaller than \\(n\\), corresponding to the set \\(\\{0, 1, \\dots, n-1\\}\\).

\n\n\n

Context: Last week I had the pleasure of visiting UPenn, where I had many interesting discussions with various people. It was also an occasion to temporarily resume a discussion/collaboration I have with Li-Yao Xia, who is currently an intern there, and Jean-Philippe Bernardy, about testing polymorphic programs and its relation to canonical representations for System F.

\n\n

During one of our blackboard discussion, Li-Yao and I did a manual proof of a cool result: we proved a parametricity theorem for \\(∀α. \\List α → \\List α\\) using syntactic methods, namely proof search among canonical proofs. (This is an idea that I have been playing with since the last year of my PhD thesis, where I unsuccessfully tried to extend my work on canonical forms for the simply-typed lambda-calculus to polymorphism. It is here worked out on an specific example, but my end goal is to turn the manual reasoning into an algorithm.)

\n\n

You may wonder, first, why the isomorphism holds. The idea is that a polymorphic function of type \\(\\List α → \\List α\\) cannot inspect the elements of the input list; it can only use them in the resulting list, possibly duplicating, reordering or dropping some elements. On any input list of size \\(n\\), the behavior of the function can be described by a list of indices in \\([0; n-1]\\). For example, if the input \\([x, y, z]\\) (for some values of \\(x, y, z\\)) gives the output \\([y, y, x]\\), then this relation will hold on any value of \\(x, y, z\\), as the function cannot inspect their value or even test them for equality. The behavior of this function on lists of size 3 can be fully described by the list of indices \\([1, 1, 0]\\). Its whole behavior is then uniquely determined by one such list for each possible size:

\n\n

\\[\n ∀α. \\List α → \\List α \\quad≃\\quad Π(n:\\Nat{}). \\List{(\\Fin n)}\n\\]

\n\n

The idea behind the “syntactic” (proof-theoretic?) proof method is the following: the set of closed values at a type \\(A\\) is isomorphic to the search space for canonical/normal derivations of \\(\\judge{}{A}\\). We have tools (in particular the notion of invertible inference rules) to reason on those – in this post I will only present the reasoning informally, but it can easily be made formally precise.

\n\n

We start by looking at the shape of the search space for

\n\n

\\[\n \\judge{}{∀α. \\List α → \\List α}\n\\] or, said otherwise, of the judgment \\[\n \\judge{}{\\List α → \\List α}\n\\]

\n\n

with a fresh/abstract type variable \\(α\\). (I will not be keeping opened type variables in context to avoid confusing them with hypotheses.)

\n\n

Any derivation of a function type, without loss of generality (w.l.o.g), is equivalent to a derivation starting with a function introduction. This is the η-expansion rule for functions: any proof term \\(e\\) is equivalent to \\(λx.~(e~x)\\), a proof that starts with a \\(λ\\). So any proof can be taken to start as follows: \\[\n\\rule{\n\\judge{\\List \\alpha}{\\List \\alpha}\n}{\n\\judge{}{\\List \\alpha \\to \\List \\alpha}\n}\n\\] we can, w.l.o.g, unfold the recursive type in the context (\\(\\List α = 1 + (α × \\List α)\\)): \\[\n\\rule{\n\\judge{1 + (α × \\List α)}{\\List α}\n}{\n\\rule{\n\\judge{\\List α}{\\List α}\n}{\n\\judge{}{\\List α → \\List α}\n}}\n\\]

\n\n

A derivation with a sum type as hypothesis can, w.l.o.g, be assumed to start by splitting on this pair (this is the η-expansion rule for sums): \\[\n\\rule{\n\\judge{1}{\\List α}\n\\quad\n\\judge{α × \\List α}{\\List α}\n}{\n\\rule{\n\\judge{1 + (α × \\List α)}{\\List α}\n}{\n\\rule{\n\\judge{\\List α}{\\List α}\n}{\n\\judge{}{\\List α → \\List α}\n}}}\n\\]

\n\n

In the right subgoal, we can always, w.l.o.g, split a hypothesis of product type: \\[\n\\rule{\n\\emptyrule{\\judge{1}{\\List α}}\n\\quad\n\\rule{\n\\judge{α, \\List α}{\\List α}\n}{\n\\judge{α × \\List α}{\\List α}\n}}{\n\\rule{\n\\judge{1 + (α × \\List α)}{\\List α}\n}{\n\\rule{\n\\judge{\\List α}{\\List α}\n}{\n\\judge{}{\\List α → \\List α}\n}}}\n\\]

\n\n

Now, an interesting pattern emerges. In the process of trying to prove \\(\\judge{\\List α}{\\List α}\\), we have to prove the (right) subgoal \\(\\judge{α,\\List α}{α}\\). We can generalize this derivation by assuming that we start with some number \\(n\\) of variables of type \\(α\\) in the context (we write \\(α^n\\) for this): \\[\n\\rule{\n\\rule{\n\\judge{\\alpha^n}{\\List \\alpha}\n}{\n\\judge{\\alpha^n, 1}{\\List \\alpha}\n}\n\\quad\n\\rule{\n\\judge{\\alpha^{n+1}, \\List \\alpha}{\\List \\alpha}\n}{\n\\judge{\\alpha^n, \\alpha \\times \\List \\alpha}{\\List \\alpha}\n}}{\n\\rule{\n\\judge{\\alpha^n, 1 + (\\alpha \\times \\List \\alpha)}{\\List \\alpha}\n}{\n\\judge{\\alpha^n, \\List \\alpha}{\\List \\alpha}\n}}\n\\]

\n\n

\\[\n\\newcommand{\\llbracket}{〚}\n\\newcommand{\\rrbracket}{〛}\n\\newcommand{\\sem}[1]{\\llbracket{} #1 \\rrbracket{}}\n\\]

\n\n

Let us write \\(\\sem{\\judge{\\alpha^n, \\List \\alpha}{\\List \\alpha}}\\) for the search space corresponding to all possible derivations of the judgment \\(\\judge{\\alpha^n, \\List \\alpha}{\\List \\alpha}\\). All the proof steps above have been done “without loss of generality” (in terms of focusing, we only used invertible rules), so they appear in any such derivation. Similarly, let us write \\(\\sem{\\judge{\\alpha^n}{\\List \\alpha}}\\) for the space of all possible derivations of \\(\\judge{\\alpha^n}{\\List \\alpha}\\), then above we have proven that \\[\n\\sem{\\judge{\\alpha^n, \\List \\alpha}{\\List \\alpha}}\n\\quad=\\quad\n\\sem{\\judge{\\alpha^n}{\\List \\alpha}}\n\\times\n\\sem{\\judge{\\alpha^{n+1}, \\List \\alpha}{\\List \\alpha}}\n\\]

\n\n

This equality can be unfolded at will \\[\n\\begin{align}\n& \\sem{\\judge{\\alpha^n, \\List \\alpha}{\\List \\alpha}} \\\\\n= & \\sem{\\judge{\\alpha^n}{\\List \\alpha}}\n \\times\n \\sem{\\judge{\\alpha^{n+1}, \\List \\alpha}{\\List \\alpha}} \\\\\n= & \\sem{\\judge{\\alpha^n}{\\List \\alpha}}\n \\times\n \\sem{\\judge{\\alpha^{n+1}}{\\List \\alpha}}\n \\times\n \\sem{\\judge{\\alpha^{n+2}, \\List \\alpha}{\\List \\alpha}} \\\\\n= & \\sem{\\judge{\\alpha^n}{\\List \\alpha}}\n \\times\n \\sem{\\judge{\\alpha^{n+1}}{\\List \\alpha}}\n \\times\n \\sem{\\judge{\\alpha^{n+2}}{\\List \\alpha}}\n \\times\n \\sem{\\judge{\\alpha^{n+3}, \\List \\alpha}{\\List \\alpha}} \\\\\n= & \\dots \\\\\n\\end{align}\n\\]

\n\n

or written as an infinite product \\[\n \\sem{\\judge{\\alpha^n, \\List \\alpha}{\\List \\alpha}}\n \\quad=\\quad\n \\prod_{k \\in \\Nat{}}{\\sem{\\judge{\\alpha^{n+k}}{\\List \\alpha}}}\n\\] and, in particular, \\[\n\\begin{align}\n& \\sem{\\judge{}{\\List \\alpha \\to \\List \\alpha}} \\\\\n= & \\sem{\\judge{\\alpha^0, \\List \\alpha}{\\List \\alpha}} \\\\\n= & \\prod_{n \\in \\Nat{}}{\\sem{\\judge{\\alpha^n}{\\List \\alpha}}} \\\\\n\\end{align}\n\\]

\n\n

Now let’s look at the structure of the derivations of \\(\\judge{\\alpha^n}{\\List \\alpha}\\). A proof of this judgment cannot start with a “left rule”, inspecting the value of one of the \\(n\\) variables of type \\(α\\), given that the structure of \\(α\\) is unknown/abstract. It must start by choosing to either build the empty list or a cons cell. We write this as follows (after unfolding the type):

\n\n

\\[\n\\rule{\n\\rule{\n\\judge{\\alpha^n}{1}\n\\quad\\oplus\\quad\n\\judge{\\alpha^n}{\\alpha \\times \\List \\alpha}\n}{\n\\judge{\\alpha^n}{1 + (\\alpha \\times \\List \\alpha)}\n}}{\n\\judge{\\alpha^n}{\\List \\alpha}\n}\n\\]

\n\n

The \\(\\oplus\\) notation between two judgments is non-standard; it means that they are not two requirements of the same proof, but two alternatives for possible proofs. All valid proofs fit that structure, and they either have a \\(\\judge{\\alpha^n}{1}\\) premise or a \\(\\judge{\\alpha^n}{\\alpha \\times \\List \\alpha}\\) premise. With this syntax, we are describing a set of possible derivations, rather than a single (partial) derivation.

\n\n

Proofs of \\(\\judge{\\Gamma}{1}\\) are trivial, and a proof of a product is always, w.l.o.g, a product of proofs (in intuitionistic logic / the λ-calculus they reuse the same context), so we can decompose further: \\[\n\\rule{\n\\rule{\n\\rule{\n}{\n\\judge{\\alpha^n}{1}\n}\n\\quad\\oplus\\quad\n\\rule\n{\n\\judge{\\alpha^n}{\\alpha}\n\\quad\n\\judge{\\alpha^n}{\\List \\alpha}\n}{\n\\judge{\\alpha^n}{\\alpha \\times \\List \\alpha}\n}\n}{\n\\judge{\\alpha^n}{1 + (\\alpha \\times \\List \\alpha)}\n}}{\n\\judge{\\alpha^n}{\\List \\alpha}\n}\n\\]

\n\n

There is exactly one possible proof of \\(\\judge{\\alpha^n}{1}\\), so its search space is \\(1\\), the unit set (with a single element). There are exactly \\(n\\) possible proofs of \\(\\judge{\\alpha^n}{\\alpha}\\), so the search space is just \\(n\\), seen as a set, or, in type-theoretic notation, \\(\\Fin{n}\\). We thus have the recursive equation: \\[\n\\sem{\\judge{\\alpha^n}{\\List \\alpha}}\n\\quad=\\quad\n1 + (\\Fin n \\times \\sem{\\judge{\\alpha^n}{\\List \\alpha}})\n\\]

\n\n

This type is either \\(1\\), or a \\(\\Fin{n}\\) and itself, recursively. This is exactly a list: \\[\n\\sem{\\judge{\\alpha^n}{\\List \\alpha}}\n\\quad=\\quad\n\\List{(\\Fin{n})}\n\\]

\n\n

so, plugging everything together: \\[\n\\begin{align}\n& \\sem{\\forall \\alpha. \\List \\alpha \\to \\List \\alpha} \\\\\n= & \\prod_{n \\in \\Nat{}}{\\sem{\\judge{\\alpha^n}{\\List \\alpha}}} \\\\\n= & \\prod_{n \\in \\Nat{}}{\\List{(\\Fin{n})}} \\\\\n\\end{align}\n\\]

\n\n

Post Scriptum

\n\n

Some of reasoning steps above can be formulated in a way that is less clear but more familiar, as a sequence of type isomorphisms. For example, the first part on \\(\\sem{\\judge{\\alpha^n, \\List\n\\alpha}{\\List \\alpha}}\\) can written as:

\n\n

\\[\n\\begin{align}\n&\n∀α. αⁿ × \\List α → \\List α\n\\\\ &\n= \\text{(unfold List)}\n\\\\ &\n ∀α. αⁿ × (1 + α × \\List α) → \\List α\n\\\\ &\n = \\text{(distribute × over +)}\n\\\\ &\n ∀α. ((αⁿ × 1) + (αⁿ⁺¹ × \\List α)) → \\List α\n\\\\ &\n = \\text{(A × 1 ≃ A)}\n\\\\ &\n ∀α. (αⁿ + (αⁿ⁺¹ × \\List α)) → \\List α\n\\\\ &\n = \\text{(A+B) → C ≃ (A→C)×(B→C)}\n\\\\ &\n ∀α. (αⁿ → \\List α) × (αⁿ⁺¹ × \\List α → \\List α)\n\\\\ &\n = \\text{(distribute ∀α below product)}\n\\\\ &\n (∀α. αⁿ → \\List α) × (∀α. αⁿ⁺¹ × \\List α → \\List α)\n\\\\\n\\end{align}\n\\]

\n\n

Reading this equational sequence, it may look like we had to make the right choice at each step; but the proof-search perspective reveals that there were in fact no choices, as each time we apply invertible rules (“w.l.o.g. rules”).

\n\n

Furthermore, some parts cannot be derived in this style; in the latter part of the proof, the isomorphism between \\(∀\\alpha. \\alpha^n → \\alpha\\) and \\(\\Fin{n}\\), which is immediate from a proof search perspective, cannot be justified in this way. (In particular, \\(A×B → C\\) is not isomorphic to \\((A→C)+(B→C)\\).)

\n\n

Going further

\n\n\n\n

You might also like

\n\n")) ((? . 77) f post (u . "The Behavior of Gradual Types: A User Study") (? . 77) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/index.html" . unix) (u . "/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/") (u . "2018-12-11T19:50:33") (? . 63) (? . 76) (c (u . "migratory typing") c (u . "gradual typing") c (u . "extended abstract") c (u . "Author: Ben Greenman")) (? . 5) #t (u . "\n\n
\n

Note: this post is an extended abstract for the paper The Behavior of Gradual Types: A User Study by Preston Tunnell—Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, click here.

\n\n

The long-term goal of gradual typing is to build languages that offer the “best” of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are at least three competing proposals for combining a dynamically-typed language with a similar statically-typed language.

\n\n
\n

It’s an interesting situation. There are dozens of papers on the semantics of gradual types—and many claim to have developers in mind—but zero papers that ask developers what they think.

\n\n

To help inform the discussion, we recently designed a survey to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.

\n\n

The image below shows one program from the survey:

\n\n

\"Figure

\n\n

This program creates an array, passes it between typed and untyped variables, and performs write & read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (\"bye\"). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type Array(String). A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type Number. These are the three behaviors in the survey:

\n\n

\"Figure

\n\n
\n

A fourth option is to reject the assignment of an Array(String) to a variable of type Array(Number). A few participants left comments asking for this behavior. See the anonymized responses for their comments, and see the paper for why we left that behavior out.

\n\n

For each behavior, we asked for respondents’ preference along two independent dimensions:

\n\n\n\n

Combined, the dimensions lead to four possible attitudes: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.

\n\n

\"Figure

\n\n

We administered the survey to three populations — software engineers, students, and Mechanical Turk workers — and thereby collected three sets of attitudes for each question. The results for the running example are below:

\n\n

\"Figure

\n\n

The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.

\n\n
\n

Unlike the survey question, the behaviors in the results are labeled as Deep, Erasure, and Shallow. These names describe the three mixed-typed semantics.

\n\n

For this question, the software engineers (left column, green bars) mostly picked the “Dislike and Unexpected” attitude for every behavior. The students (mid column, blue bars) also show consensus on “Dislike and Unexpected” for the Deep and Erasure behaviors; however, they are split for the Shallow behavior. The Mechanical Turk workers are divided on every behavior.

\n\n

See the paper for the other questions and responses.

\n\n

Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:

\n\n

if you are designing a mixed-typed language and choose not to enforce full types, then make sure to explain this behavior to users!

\n\n

Put lots of example programs in the language’s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.

\n\n

Questions

\n\n

Here are some good questions we’ve gotten that are not clearly answered in the paper.

\n\n

Q. Did any respondents “expect” more than one behavior?

\n\n

Yes, 59% of the software engineers and 82% of the students selected “Liked and Expected” and/or “Dislike and Expected” for different behaviors on the same program.

\n\n\n\n\n\n\n

Q. Did the respondents have a prior preference for static or dynamic typing?

\n\n

Near the end of the survey we asked: “Which do you prefer, typed or untyped programming?”. See table 2 of the paper for coded responses to this question, or the anonymized responses for the ground truth. Most preferred typed programming.

")) ((? . 66) f post (u . "Trees, 1973") (? . 66) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/07/19/trees-1973/index.html" . unix) (u . "/blog/2017/07/19/trees-1973/") (u . "2017-07-19T21:48:56") (? . 9) (? . 65) (c (u . "history") c (u . "Author: Ben Greenman")) (u . "\n

From the PRL archives:

\n\n
\n

I think that I shall never see a matrix lovely as a tree. — Trees, by Guy L. Steele Jr., MIT, 1973

") #t (u . "\n

From the PRL archives:

\n\n
\n

I think that I shall never see a matrix lovely as a tree. — Trees, by Guy L. Steele Jr., MIT, 1973

\n\n\n
\n\n

You might recognize the opening line from Joyce Kilmer’s 1914 poem Trees, or from Radia Perlman’s Algorhyme (published 1985).

\n\n

The poem is online in at least one other place, but the copy linked above (from BYTE magazine) comes with a footnote on How this poem came to be printed.

")) ((? . 35) f post (u . "Disappearing Code") (? . 35) 1731622765 (p+ #"/home/runner/work/website/website/blog/2018/11/24/disappearing-code/index.html" . unix) (u . "/blog/2018/11/24/disappearing-code/") (u . "2018-11-24T09:52:58") (? . 34) (? . 33) (c (u . "dear diary") c (u . "Author: Ben Greenman")) (u . "\n

Two experiences at SPLASH 2018 reminded me that software gets thrown away and replaced.

") #t (u . "\n

Two experiences at SPLASH 2018 reminded me that software gets thrown away and replaced.

\n\n\n

Story 1

\n\n

The first reminder came near the end of a talk by Martin Rinard. Once upon a time, Martin was working as a consultant and a firm asked him to review a software package. (The firm wanted a second opinion about how the software computed its results.) The firm sent a zipfile; Martin found six versions of the code inside; the firm said “well, please check all six versions”; and it turned out:

\n\n\n\n

The moral of Martin’s story was: (1) the creators of a software system are often different from the maintainers, and (2) researchers need to build tools to help these maintainers.

\n\n

Story 2

\n\n

The second reminder came from a teaching assistant who said the functional programming course at their institution was currently using a Python script to test students’ code. Once upon a time, I was a teaching assistant for the same course at the same institution. We had trouble testing students’ code via the Python script left by the pre–2013 course staff, so I wrote a command-line tool to handle the tests and other compile/run/grade tasks. To keep history from repeating itself, I used the same language the course teaches (OCaml) and wrote some documentation — but it seems like that was not enough. At any rate, writing the tool was a good exercise.

\n\n
\n

In the end, everybody must understand for himself. — Per Martin-Löf

\n\n

Reflection

\n\n

In each story, the maintainers of a software system threw away some old code to make their job easier in the short term. How can we stop this “re-inventing the wheel” from happening?

\n\n

Martin Rinard’s solution is to let maintenance programmers keep their current habits, but provide tools to make the short-term, pragmatic solutions into a more robust systems. Search for \"failure-oblivious computing\" to learn more (this was the topic of his talk).

\n\n

In Story 1, the maintainers were able to avoid the DSL by modifying an inherited blob of DSL-generated code. If the DSL did not generate code, history might have taken a different course; it might be best to start with a language that offers tools for linguistic re-use, and to build a DSL from these tools — so there is no generated code. The Racket programming language is exploring this path. For a recent example, see the video-lang paper.

\n\n

The Story 2 test harness, however, was not generating code. Its maintainers discarded a “big” program written in a typed functional language in favor of a script. Perhaps we need a language that allows mixing statically-typed and dynamically-typed code (shouts out to my own research).

\n\n

The best solution is probably to start with a team and keep the culture alive. Always pair program!

\n\n
\n\n

Addendum: comment from Mitch Wand

\n\n
\n

The best solution is probably to start with a team and keep the culture alive. Always pair program!

\n\n

Ermm, this works better for sourdough bread than for people.

\n\n

Even in the not-so-real world of checking student solutions, there’s often no way of guaranteeing that one half of a pair will be around for the second round. They may be on co-op. Or the course will not be offered the next semster/year/etc. Or the course will change at the next offering (from OCaml to Python or from Racket to Java) so that large chunks of the infrastructure will have to be discarded or rewritten.

\n\n

The “real” solution is to write literate code (as we preached incessantly in PDP), so that the next reader will have at least some clue as about what you wrote. This just may be sufficient incentive to modify rather than rebuild from scratch.

\n\n

Ever the optimist, —Mitch

")) ((? . 47) f post (u . "NEPLS on October 7th at Northeastern University") (? . 47) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/index.html" . unix) (u . "/blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/") (u . "2016-09-15T21:18:45") (? . 46) (? . 7) (c (u . "NEPLS") c (u . "conference") c (u . "Author: Ben Greenman")) (u . "\n

The next edition of the New England Programming Language Seminar (NEPLS) will be held on Friday, October 7th at Northeastern University. Organizers are Gabriel Scherer and Max New. See you there!

") #t (u . "\n

The next edition of the New England Programming Language Seminar (NEPLS) will be held on Friday, October 7th at Northeastern University. Organizers are Gabriel Scherer and Max New. See you there!

\n\n\n

Here is the official announcement from the NEPLS mailing list.

\n\n
\n

Hi everyone,

\n

The next New England Programming Languages and Systems Symposium will take place on

\n
\n

Friday, October 7th 2016

\n

at

\n
\n

Northeastern University, Boston.

\n

Please mark it in your calendars!

\n

The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.

\n

Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.

\n

The submission deadline is

\n
\n

Sunday, September 25th.

\n

Acceptance notifications will be out on Thursday, September 28th.

\n

Send your proposal to

\n
\n

nepls-talks-committee@lists.cs.brown.edu

\n

More details about NEPLS are available on the NEPLS webpage:

\n
\n

http://www.nepls.org/

\n\n

To subscribe to the NEPLS mailing list, visit this page:

\n\n

https://lists.cs.brown.edu/sympa/subscribe/nepls

")) ((? . 84) f post (u . "Refinement Types") (? . 84) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/04/20/refinement-types/index.html" . unix) (u . "/blog/2017/04/20/refinement-types/") (u . "2017-04-20T23:38:23") (? . 85) (? . 86) (c (u . "HOPL") c (u . "Author: Kevin Clancy")) (? . 5) #t (u . "\n\n

Roughly, a refinement type system is an extra layer of precision, enforced through subtyping, added onto an existing type system. A base type is decomposed into a set of base refinements, each of which denotes a subset of the values belonging to the base type. A subtyping relation respecting set inclusion can then be defined among the refinements of the base type. These subtyping relations can be lifted onto a subtyping relation for compound types using a standard arrow subtyping rule.

\n\n

Extra type-checking precision sounds great, but what in practical terms does this precision look like? Freeman and Pfenning’s ’92 paper Refinement Types for ML proposes extending ML’s type definition language with constructs for decomposing a discriminated union type into a lattice of subtypes. For example, it allows the decomposition of a list type into a lattice including base refinements for empty lists, non-empty lists, and singletons. Those with experience in functional programming will realize this alleviates the dreaded and inescapable “non-exhaustive pattern match” warning, which tends to crop up in situations where the programmer understands that an exhaustive pattern match is not necessary.

\n\n

In the late 90’s Xi and Pfenning advanced the state of refinement types by introducing a dependent refinement type system, implemented as a tool called Dependent ML. Their approach identifies a base refinement using a tuple of terms drawn from some computationally tractable constraint language called an index language. A list datatype can then be refined with a term of the linear integer arithmetic index language, denoting the subset of all lists having a specific length. One list refinement is then considered a subtype of another when a constraint solver can prove their index terms equal. Vazou et. al.’s recent project Liquid Haskell is another dependent refinement type system which decides subtyping among base types by invoking an SMT solver under a context-dependent set of constraints. It differs significantly from Dependent ML in that it refines base types with certain well-behaved program terms rather than indices.

\n\n
\n\n

Resources:

\n\n")) ((? . 83) f post (u . "Type Inference in Stack-Based Programming Languages") (? . 83) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/03/10/type-inference-in-stack-based-programming-languages/index.html" . unix) (u . "/blog/2017/03/10/type-inference-in-stack-based-programming-languages/") (u . "2017-03-10T16:23:30") (? . 2) (? . 82) (c (u . "HOPL") c (u . "Author: Rob Kleffner")) (? . 5) #t (u . "\n\n

Stack-based languages occupy a niche in today’s programming language environment. The predominant stack-based language in use by programmers is Forth, and is found mostly on embedded devices. These languages also find use as compile targets for more popular languages: the CIL and JVM are both stack-based. Less popular but highly interesting languages to mention include Joy and Factor, known for their emphasis on higher-order stack-based programming.

\n\n

The majority of stack-based languages are not statically typed, and it would be a stretch to call Forth even dynamically typed. As such, developing large projects in Forth or Factor can require great discipline on the part of the programmer to avoid type errors.

\n\n

In this talk, I presented the development of type inference for stack-based languages as a linear sequence, divided into two overarching segments:

\n\n\n\n

The thread of research on stack effects began with Jaanus Pöial in the early 1990’s, and is a formalization of a commenting style well-known in the Forth community. The nested tuple systems were first examined by Okasaki in 1993 in the context of Haskell, and were later applied to higher-order stack-based languages. At the end, I give some avenues for extending the research on these systems, and list some pitfalls to be avoided in further research.

\n\n

Full notes (as PDF documents) — see the git repository for more documents:

\n\n")) ((? . 73) f post (u . "The Typed Racket Optimizer vs. Transient") (? . 73) 1731622765 (p+ #"/home/runner/work/website/website/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/index.html" . unix) (u . "/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/") (u . "2020-01-15T12:16:35") (? . 64) (? . 72) (c (u . "typed racket") c (u . "transient") c (u . "offsite") c (u . "Author: Ben Greenman")) (u . "\n

What type-directed optimizations does Typed Racket perform and do any require full types?

") #t (u . "\n

What type-directed optimizations does Typed Racket perform and do any require full types?

\n\n\n
\n

This post is based on a short talk. Slides from the talk are here: http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf

\n\n

Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [JFP 2019]. And so, the standard types are very strong but (possibly) slow.

\n\n

Lately, I’ve been working on a transient back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee — only that typed code cannot get “stuck” — via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [ICFP 2018], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.

\n\n

The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:

\n\n
    \n
  1. a static type checker,
    2. \n
  2. a compiler from types to contracts, and
    3. \n
  3. a type-driven optimizer [PADL 2012, OOPSLA 2012].
\n\n

Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?

\n\n

Q. Can Transient re-use the Typed Racket optimizer?

\n\n

The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.

\n\n\n\n

The idea of a “shape” is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.

\n\n

Here are a few shapes according to the current Transient prototype:

\n\n
  Shape(Natural)                = Natural\n  Shape(Listof String)          = Listof Any\n  Shape(Symbol -> Boolean)      = Any -> Any\n  Shape(Vector Void Void)       = Vector Any Any\n  Shape(U Void (Listof Symbol)) = U Void (Listof Any)
\n\n

For the current shapes, can we re-use the Typed Racket optimizer?

\n\n

Optimization Topics

\n\n

Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of “safe” or “unsafe” for Transient.

\n\n

To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.

\n\n
\n\n

Topic 1: apply

\n\n

apply.rkt “inlines” expressions of the form (apply f (map g xs)) to map and fold in one pass over the list (xs). Currently, the pass only triggers when f is + or *.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: xs (Listof Integer))\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (apply + (map abs xs))\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (let loop ((v 0)\n             (lst xs))\n    (if (null? lst)\n      v\n      (loop (+ v (abs (unsafe-car lst)))\n            (unsafe-cdr lst))))
\n\n

Verdict: safe, but risky.

\n\n

Technically, this transformation is unsound for Transient because of how it uses unsafe-car. The expansion of (apply * (map g xs)) applies (g (unsafe-car xs)) without confirming that the first element of xs matches its expected type. This unsoundness is no problem, though, as long as every Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)

\n\n
\n\n

Topic 2: box

\n\n

box.rkt safely applies unsafe box operations to expressions with Box type.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: b (Boxof Symbol))\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (unbox b)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-unbox b)
\n\n

Verdict: safe

\n\n
\n\n

Topic 3: dead-code

\n\n

dead-code.rkt uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with if expressions, lambda expressions, and case-lambda; the latter is the most interesting for Transient.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: f (-> Symbol Symbol)\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (define f\n    (case-lambda\n      ((s) s)\n      ((s i)\n       (for/list ((_i (in-range i))) s))))\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (define f\n    (case-lambda\n      ((s) s)\n      ((s i)\n       ; dead code, replace with no-op\n       (void))))
\n\n

Verdict: unsafe, can change behavior

\n\n

The pass infers that some branches of a case-lambda can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the “untyped” branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to (void) and thereby violate the graduality design goal [SNAPL 2015, ICFP 2018] — that is, that adding types should only change behavior by introducing runtime type mismatches.

\n\n
\n\n

Topic 4: extflonum

\n\n

extflonum.rkt safely applies unsafe extflonum operations to expressions with Extflonum type.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: e Extflonum)\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (extflabs e)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-extflabs e)
\n\n

Verdict: safe

\n\n
\n\n

Topic 5: fixnum

\n\n

fixnum.rkt safely applies unsafe fixnum operations to expressions with Fixnum type.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: f Fixnum)\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (exact->inexact f)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-fx->fl f)
\n\n

Verdict: safe

\n\n
\n\n

Topic 6: float-complex

\n\n

float-complex.rkt unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: f (-> Float-Complex Float-Complex Float-Complex))\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (define (f n0 n1)\n    (+ n0 n1))\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (define (f n0 n1)\n    (let* ((unboxed-real-0 (unsafe-flreal-part n0))\n           (unboxed-imag-0 (unsafe-flimag-part n0))\n           (unboxed-real-1 (unsafe-flreal-part n1))\n           (unboxed-imag-1 (unsafe-flimag-part n1))\n           (unboxed-real-2 (unsafe-fl+ (real->double-flonum unboxed-real-0)\n                                       unboxed-real-1))\n           (unboxed-imag-2 (unsafe-fl+ (real->double-flonum unboxed-imag-0)\n                                       unboxed-imag-1)))\n      (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))
\n\n

Verdict: safe, with caution

\n\n

The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the float-complex pass creates functions that apply unsafe-flreal-part before anything else; to be safe, the pass needs to make sure that Transient checks come first.

\n\n
\n\n

Topic 7: float

\n\n

float.rkt safely applies unsafe flonum operations to expressions with Flonum type and also transforms some random calls to use unsafe-flrandom.

\n\n

Example

\n\n
  ;; --------------------------------------------------\n  ;; Before Optimization\n  (random)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-flrandom (current-pseudo-random-generator))
\n\n

Verdict: safe, but a close call

\n\n

Accessing a parameter, as in (current-pseudo-random-generator), is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of unsafe-flrandom.

\n\n
\n\n

Topic 8: list

\n\n

list.rkt safely applies unsafe list operations to list expressions.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: lst (List Symbol Symbol))\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (list-ref lst 0)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-list-ref lst 0)
\n\n

Verdict: safe, with strong-enough shape checks

\n\n

The shape check for a (Listof T) must check for proper lists (via list?); note that the cost of this check depends on the size of incoming values. The shape check for a (List T ...) type must validate the length of incoming values.

\n\n
\n\n

Topic 9: number

\n\n

number.rkt performs simple transformations on Real-valued expressions.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: r Real)\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (+ r)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  r
\n\n

Verdict: safe

\n\n
\n\n

Topic 10: pair

\n\n

pair.rkt safely applies pair-access operations to (possibly-nested) pairs.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: p (Pairof (Pairof Symbol Void) String))\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (cdar p)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-cdr (unsafe-car p))
\n\n

Verdict: unsafe

\n\n

Transient guarantees the first level of a type, but nothing more. Concretely, Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any and so the unsafe-cdr above is not safe.

\n\n
\n\n

Topic 11: sequence

\n\n

sequence.rkt safely applies unsafe sequence operations to expressions with (Sequenceof T) type.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: s String)\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (for ((c s))\n    (void))\n\n  ;; --------------------------------------------------\n  ;; After Optimization (simplified)\n  (for ((c (in-string s)))\n    (void))
\n\n

Verdict: safe, with strong enough shape checks (see list and vector)

\n\n
\n\n

Topic 12: string

\n\n

string.rkt safely applies unsafe string operations to expressions with String type. (Note that unsafe-string-ref is only safe when the result is sure to be a Latin–1 character.)

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: b Bytes)\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (bytes-length b)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-bytes-length b)
\n\n

Verdict: safe

\n\n
\n\n

Topic 13: struct

\n\n

struct.rkt safely applies unsafe struct operations to struct expressions, using Typed Racket’s internal registry of struct info.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (struct open-interval ([lo : Real] [hi : Real]))\n  (: ivl open-interval)\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (open-interval-lo ivl)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-struct-ref ivl 0)
\n\n

Verdict: safe

\n\n
\n\n

Topic 14: unboxed-let

\n\n

unboxed-let.rkt cooperates with the float-complex pass by transforming the binding-site of some complex numbers. This pass may change a let-expression into a let-values that expects a real-part and imag-part, and may change a function to expect twice as many arguments — provided the optimizer can find all calls to the function.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: k Float-Complex)\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (let ((f (lambda ((n : Float-Complex)) (+ n n))))\n    (f k))\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (let ((f (lambda (real-part-n imag-part-n) ....)))\n    (f (unsafe-flreal-part k) (unsafe-flimag-part k)))
\n\n

Verdict: safe, thanks to the (conservative) escape analysis

\n\n
\n\n

Topic 15: vector

\n\n

vector.rkt safely applies vector operations to vector expressions.

\n\n

Example

\n\n
  ;; Type Assumptions\n  (: v (Vector (Listof Symbol) String))\n  (: lst (Listof Symbol))\n\n  ;; --------------------------------------------------\n  ;; Before Optimization\n  (vector-set! v lst 0)\n\n  ;; --------------------------------------------------\n  ;; After Optimization\n  (unsafe-vector-set! v lst 0)
\n\n

Verdict: safe, with strong-enough shape checks

\n\n

The check for (Vector T ...) must check the length of incoming values.

\n\n
\n\n

Summary

\n\n

The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (dead-code, pair). One must take care when rewriting a Transient function (float-complex). One may limit our ability to reduce the number of run-time checks in a program (apply). Two others require transient checks whose cost depends on the size of the input values (list, sequence).

\n\n

There may be other issues that I missed while reading the optimizer code. If so, I’ll try to remember to update this post.

")) ((? . 42) f post (u . "SRC-submissions") (? . 42) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/11/17/src-submissions/index.html" . unix) (u . "/blog/2016/11/17/src-submissions/") (u . "2016-11-17T13:52:52") (? . 41) (? . 36) (c (u . "Author: Gabriel Scherer")) (u . "\n

Max New, Daniel Patterson and Ben Greenman recently wrote three two-page abstracts on what they are working on right now. Come have a look — and any feedback is welcome!

") #t (u . "\n

Max New, Daniel Patterson and Ben Greenman recently wrote three two-page abstracts on what they are working on right now. Come have a look — and any feedback is welcome!

\n\n\n

Gradual Type Precision as Retraction

\n\n

Gradual Type Precision as Retraction\n
Max New\n
2016

\n\n
\n

Gradually typed programming languages allow for a mix of precision of static type information, allowing advanced type features to be added to existing languages, while still supporting interoperability with legacy code. The advantages of gradual typing are enticing to researchers and practitioners alike, but a general theory of gradually typed languages is only beginning to emerge after a decade of research.

\n

It has long been noted that there is much similarity between work on contracts and gradual typing, and the use of retracts in domain theory which were used to relate models of untyped and typed lambda calculus in Scott(1976) and Scott(1980). Here we take this connection seriously and consider how judgments in modern gradually typed languages can be framed in terms of retractions. While retractions in programming languages were originally studied in terms of denotational semantics in domains, our presentation will use only the most basic elements of category theory: composition, identity and equality of terms, so our formulation is equally applicable to axiomatic or operational semantics.

\n

In particular we propose a semantic criterion for the notion of precision of gradual types, a common judgment in gradually typed languages (sometimes called naive subtyping for historical reasons). We relate it to a previous definition from Wadler and Findler(2009) that defines type precision in terms of blame. We show that our definition decomposes in a similar way into “positive” and “negative” type precision, but without depending on a specific notion of blame in the language.

\n\n

Linking Types: Specifying Safe Interoperability and Equivalences

\n\n

Linking Types: Specifying Safe Interoperability and Equivalences\n
Daniel Patterson\n
2016

\n\n
\n

All programs written in high-level languages link with libraries written in lower-level languages, often to expose constructs, like threads, random numbers, or automatic serialization, that aren’t possible in the high-level language. This linking usually takes place after compiling both languages to a common language, possibly assembly. In this sense, reasoning about crosslanguage linking means reasoning about compilation.

\n

While most languages include cross-language linking (FFI) mechanisms, they are ad-hoc and can easily break the semantic equivalences of the source language, making it hard for source programmers to reason about correctness of their programs and hard for compiler writers to reason about correctness of their optimizations.

\n

In this work, I design and motivate linking types, a language-based mechanism for formally specifying safe linking with libraries utilizing features inexpressible in the source. Linking types allows programmers to reason about their programs in the presence of behavior inexpressible in their language, without dealing with the intricacies of either the compiler or the particular language they are linking with.

\n\n

Pruning Contracts with Rosette

\n\n

Pruning Contracts with Rosette\n
Ben Greenman\n
2016

\n\n
\n

Contracts are a pragmatic tool for managing software systems, but programs using contracts suffer runtime overhead. If this overhead becomes a performance bottleneck, programmers must manually edit or remove their contracts. This is no good. Rather, the contracts should identify their own inefficiencies and remove unnecessary dynamic checks. Implementing contracts with Rosette is a promising way to build such self-aware contracts.

\n\n

While we’re at it let’s rant on SRCs

\n\n

These abstracts are submitted at POPL’s “Student Research Competition”. You submit an abstract, and if you get accepted to that thing, you get a bit of travel support money, and you have to prepare a poster and present it at the conference.

\n\n

I have a firm dislike for the Competition part of that concept: I think that people think of research too competitively already, and that we should have less of that, not more. (Having some is unfortunately unavoidable in scarce-resource situations.) I think that the process of awarding prizes to students with the “best poster” is dumb — and borderline ridiculous.

\n\n

On the other hand, my experience seeing them writing these extended abstracts is that it’s a useful exercise for them, and produces nice result — short, readable introductions to their ideas. And Jennifer Paykin convincingly argues that although writing a poster is rather painful, actually presenting it during the conference is interesting and useful. In her words, “it’s worth it to get the experience of authentic and fruitful discussions”. Plus having posters in the corridors of one’s lab is very nice.

\n\n

I think we could have “Student Research Sessions” or “Student Poster Sessions”, where students are encouraged to present their work, would write those nice extended abstracts and posters, interact with researchers at the conference, and get travel money, without the ranking and prize stuff. (I would still encourage students to participate to SRC today, it seems to be worth it.)

")) ((? . 51) f post (u . "Building a Website with Scribble") (? . 51) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/05/23/building-a-website-with-scribble/index.html" . unix) (u . "/blog/2017/05/23/building-a-website-with-scribble/") (u . "2017-05-23T01:53:13") (? . 22) (? . 50) (c (u . "Scribble") c (u . "tutorial") c (u . "Author: Ben Greenman")) (u . "\n

The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!

") #t (u . "\n

The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!

\n\n\n

The Story so Far

\n\n

Last Fall, I took a flight to Chicago (on my way to RacketCon 2016). When I landed, there was a new message in my inbox:

\n\n
    Subject: Web Page\n    Date: 2016-09-15\n\n    You have been nominated webmaster by public acclamation. Congratulations!
\n\n

Emboldened by the trust of my people, I promptly converted the PRL website from Racket-generating-HTML to the fine scribble/html preprocessor language (commit a0600d) This bold action polarized the community.

\n\n
\n

I can’t read the source anymore! Is this really an improvement?

\n\n

Fear not, citizens. The switch to scribble/html was the right choice, and you too can learn to read the source code.

\n\n

How to Read scribble/html Programs

\n\n

Basics

\n\n

Scribble is a language for writing Racket documentation. The key innovation in Scribble is the @-expression (read: “at expression”). The scribble/html language combines @-expression syntax with functions that generate HTML.

\n\n

@-syntax

\n\n

Greg Hendershott and the Scribble Documentation explain @-expressions properly. Here’s a short tutorial (Part 1 of 2, “the basics”):

\n\n\n\n

Examples: Evaluating \"Hello Dave\" puts “Hello Dave” in your document. Evaluating \"Hello @Dave\" puts “Hello ???” in your document, where \"???\" is the value of the variable Dave. Finally if Dave is the name of a function, then \"Hello @(Dave)\" calls the Dave function with zero arguments and puts whatever it returns into your document.

\n\n

To make it easy to interleave text, function calls, and code, Scribble discriminates between 4 kinds of parentheses when they follow an @-sign (Part 2 of 2, “the parens”):

\n\n\n\n

“Unescapable text mode” treats @-signs as text instead of toggling between modes.

\n\n

Generating HTML

\n\n

The scribble/html language comes with functions that render HTML. These functions have the same name as the corresponding HTML tag.

\n\n

Example program:

\n\n
\n \n \n \n \n
\n
\n 
1\n2
\n
\n 
#lang scribble/html\n@p{Hello World}\n
\n
\n
\n\n

Running this program prints:

\n\n
\n \n \n \n \n
\n
\n 
1
\n
\n 
<p>Hello World</p>\n
\n
\n
\n\n

No surprises.

\n\n

One thing that is surprising is how scribble/html handles tag attributes. Every tag-rendering function accepts “Racket mode” arguments that specify an attribute name and attribute value.

\n\n

For example:

\n\n
\n \n \n \n \n
\n
\n 
1\n2
\n
\n 
#lang scribble/html\n@p[style: \"color:red\"]{Hello World}\n
\n
\n
\n\n

Prints:

\n\n
\n \n \n \n \n
\n
\n 
1
\n
\n 
<p style=\"color:red\">Hello World</p>\n
\n
\n
\n\n

Hope the output looks familiar. The input syntax is strange, but that’s what it is.

\n\n

Larger programs print larger webpages. Each page on the PRL website is HTML generated by one scribble/html program.

\n\n

Why scribble/html is an Improvement

\n\n

Before scribble/html, the PRL website was implemented in scribble/text. A scribble/text program renders and prints text. There is no extra support for HTML.

\n\n

To compare, here’s the start of the old homepage:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7\n8\n9
\n
\n 
#lang scribble/text\n@(require \"templates.rkt\")\n\n<!DOCTYPE html>\n<html lang=\"en\">\n  @(header \"Home\")\n  <body id=\"pn-top\">\n    @(navbar \"Home\")\n    <div class=\"jumbotron\">\n
\n
\n
\n\n

And here is the start of the scribble/html’d homepage:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7\n8\n9
\n
\n 
#lang scribble/html\n@require[\"templates.rkt\"]\n\n@doctype{html}\n@html[lang: \"en\"]{\n  @header{Home}\n    @body[id: \"pn-top\"]{\n      @navbar{Home}\n      @div[class: \"jumbotron\"]{\n
\n
\n
\n\n

The pages look similar. The new one has more @-signs and parentheses, the old one has more <-signs and quotes. If you were able to edit the old page, you should be able to edit the new page.

\n\n

The key improvement in the new page is that common mistakes are now compile-time errors.

\n\n\n\n

Both flavors of error message come with source-code line numbers. This is very very helpful.

\n\n

Small Improvements

\n\n

1. More Functions

\n\n

Before, the Teaching page contained some interesting HTML for rendering vertical text (look for the word “Semantics” to see how this was used):

\n\n
\n \n \n \n \n
\n
\n 
1
\n
\n 
<span class=\"how-to-design-programs\">S<br />e<br />m<br />a<br />n<br />t<br />i<br />c<br />s<br /><br /></span>\n
\n
\n
\n\n

After, the same text is generated from a function call:

\n\n
\n \n \n \n \n
\n
\n 
1
\n
\n 
@span[class: \"how-to-design-programs\"]{@vertical-text{Semantics}}\n
\n
\n
\n\n

The vertical-text function is simple:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4
\n
\n 
@require[(only-in racket/list add-between)]\n\n@(define (vertical-text . str*)\n   (add-between (string->list (append* str*)) (br)))\n
\n
\n
\n\n

2. More Structure, Less Boilerplate

\n\n

Here’s part of the old definition of “Ben Greenman” on the People page:

\n\n
\n \n \n \n \n
\n
\n 
 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23
\n
\n 
<div class=\"row pn-person\">\n  <div class=\"col-md-12 pn-row-eq-height\">\n    <div class=\"col-md-3 pn-photo\">\n      <div class=\"img-wrapper\">\n        <img src=\"img/ben_greenman.jpg\" title=\"Ben Greenman\" alt=\"Ben Greenman\" />\n      </div>\n    </div>\n    <div class=\"col-md-9\">\n      <div class=\"col-md-4 pn-contact\">\n        <span class=\"pn-name\">Ben Greenman</span><br />\n        Advisor: Matthias Felleisen<br />\n        <a href=\"mailto:types@\"@\"ccs.neu.edu\">types@\"@\"ccs.neu.edu</a><br />\n        <a href=\"http://www.ccs.neu.edu/home/types\">www.ccs.neu.edu/home/types</a>\n      </div>\n      <div class=\"col-md-3 pn-muted col-md-offset-5\">\n        Joined 2014\n      </div>\n      <div class=\"col-md-12 pn-bio\">\n        <p>I like constructions .... </p>\n      </div>\n    </div>\n  </div>\n</div>\n
\n
\n
\n\n

The new definition uses a helper function with keyword arguments for each “field” of the person:

\n\n
\n \n \n \n \n
\n
\n 
1\n2\n3\n4\n5\n6\n7\n8
\n
\n 
@person[#:name \"Ben Greenman\"\n        #:title \"Advisor: Matthias Felleisen\"\n        #:e-mail \"types@ccs.neu.edu\"\n        #:website \"http://ccs.neu.edu/home/types\"\n        #:history @list[\"Joined 2014\"]\n        #:img \"ben_greenman.jpg\"]{\n  I like constructions ....\n}\n
\n
\n
\n\n

3. Less String-Formatting

\n\n

Before, the code did a lot of string formatting (link):

\n\n
\n \n \n \n \n
\n
\n 
1\n2
\n
\n 
(define (link url body)\n  (string-append \"<a href=\\\"\" url \"\\\">\" body \"</a>\"))\n
\n
\n
\n\n

The new code has no need for such helper functions.

\n\n
\n \n \n \n \n
\n
\n 
1
\n
\n 
@a[href: url body]\n
\n
\n
\n\n

Bottom Line

\n\n

Scribble is a good language for making static HTML pages.

\n\n
\n\n

If you liked this post, you may also be interested in:

\n\n")) ((? . 56) f post (u . "Emacs daemon for fast editor startup") (? . 56) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/index.html" . unix) (u . "/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/") (u . "2016-10-17T21:48:25") (? . 7) (? . 55) (c (u . "System Administration") c (u . "Emacs") c (u . "Author: Gabriel Scherer")) (u . "\n

In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs’ bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors — on each git commit for example, or when I use a Firefox extension to edit a textarea content in a proper editor.

\n\n

In this blog post, I describe how to use emacsclient to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.

") #t (u . "\n

In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs’ bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors — on each git commit for example, or when I use a Firefox extension to edit a textarea content in a proper editor.

\n\n

In this blog post, I describe how to use emacsclient to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.

\n\n\n

Emacs has long supported a client/server mode: a daemon emacs instance is loaded in the background, and whenever you request a new emacs window you can creates a new frame (~ window) using the same instance. This means that the startup time is dramatically reduced after the daemon is launched, as for example the execution of your personal configuration code does not have to be repeated.

\n\n

To use it, I have this code as /usr/bin/editor:

\n\n
\n \n \n \n \n
\n
\n 
1\n2
\n
\n 
#!/bin/bash\nemacsclient -a \"\" -c \"$@\"\n
\n
\n
\n\n

The empty -a parameter means that if no daemon exists, it should start one in the background and retry. The -c option means that a new frame (window) should be created instead of reusing an existing one. \"$@\"means that when the script is invoked with a path as command-line parameter (editor /tmp/foo.txt), the corresponding file will be opened.

\n\n

Finally, my .bash_profile sets the EDITOR variable to editor (export EDITOR=/usr/bin/editor); this environment variable is what most tools (git included) will use to invoke a text editor.

\n\n

On my machine, starting the daemon takes 1.4s. Creating a client windows takes around 0.150s.

\n\n

If you want to control the environment in which the daemon process is started, you can launch it explicitly by running emacs --daemon.

\n\n

Cool kids use Spacemacs these days, which comes with all sort of convenient settings built in, and I’m told that it does daemonization out of the box. I haven’t taken the time to give Spacemacs a try yet.

\n\n

Finally, sometimes having all editor windows share the same process is not the right thing, because I do stuff which makes Emacs a bit unstable. (It’s not very good at asynchronous communication with the rest of the world, so for example accessing a file through SSH from Emacs can hang the process when network goes bad.). I’ve been bitten a few times by a crash of all editor windows at the same time, and since then, when I know I’m going to do “heavy stuff”, I launch a separate process for it (just emacs instead of editor or emacsclient).

\n\n

P.S.: To precisely measure the startup time, ask Emacs to evaluate a Lisp expression on startup, to kill it immediately.:

\n\n
\n \n \n \n \n
\n
\n 
1\n2
\n
\n 
$ time emacs --eval \"(save-buffers-kill-terminal)\"\n$ time emacsclient -a '' -c -e \"(save-buffers-kill-terminal)\"\n
\n
\n
")) ((? . 17) f post (u . "Monotonicity Types: Towards A Type System for Eventual Consistency") (? . 17) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/index.html" . unix) (u . "/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/") (u . "2017-10-22T11:59:06") (? . 16) (? . 4) (c (u . "types") c (u . "monotonicity") c (u . "CRDTs") c (u . "eventual consistency") c (u . "Author: Kevin Clancy")) (u . "\n

A few weeks back, we published a draft of an article entitled Monotonicity Types. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.

") #t (u . "\n

A few weeks back, we published a draft of an article entitled Monotonicity Types. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.

\n\n\n

But first, what, precisely, do we mean by monotonicity? Here’s a short definition:

\n\n

A partially ordered set is a set \\(P\\) endowed with a relation \\(\\leq\\) such that for all \\(p, q, r \\in P\\) we have:

\n\n
    \n
  1. \\(p \\leq p\\) (reflexivity)
    2. \n
  2. \\(p \\leq q\\) and \\(q \\leq r\\) implies \\(p \\leq r\\) (transitivity)
    3. \n
  3. \\(p \\leq q\\) and \\(q \\leq p\\) implies \\(p = q\\) (anti-symmetry)
\n\n

If \\(P\\) and \\(Q\\) are partially ordered sets, we say that a function \\(f : P \\to Q\\) between them is monotone if for all \\(p_1, p_2 \\in P\\) with \\(p_1 \\leq p_2\\), we have \\(f(p_1) \\leq f(p_2)\\).

\n\n

So, said another way, increasing the input to a monotone function causes an increase to its output.

\n\n

Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [Meiklejohn et al. (2015)] and BloomL [Conway et al. (2012)], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [Shapiro et al. (2011)] and LVars [Kuper et al. (2013)] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.

\n\n

Thus if a user would like to make use of such a language for concurrent and distributed programming, they’re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.

\n\n

To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let’s look at an example. Suppose we need a function to determine whether a replicated counter’s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter’s value:

\n\n
fun IsOdd(x : Nat) = x % 2 == 1
\n\n

However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can’t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.

\n\n
fun MoreThanTen(x : Nat) = x > 10
\n\n

The boolean values \\(true\\) and \\(false\\) form one of the simplest partially ordered sets of all. We consider \\(false \\leq false\\), \\(false \\leq true \\), and \\( true \\leq true \\). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \\(x > 10\\) to flip from false to true, but not vice versa. When we observe that the local counter replica P’s value is greater than 10, we don’t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P’s current value. This is where MoreThanTen’s monotonicity becomes useful. At the point when all replicas have received P’s current information, every replica in the system will agree that MoreThanTen applied to the counter’s value returns true.

\n\n

We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.

\n\n

Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an sfun, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.

\n\n

Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone separately in each of its arguments; that is, for \\(i \\in 1..n\\), if \\(p_i \\leq p_i'\\) then \\(f(p_1, \\ldots, p_i, \\ldots, p_n) \\leq f(p_1, \\ldots p_i', \\ldots p_n)\\).

\n\n

The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:

\n\n

1.) plus : \\( (x : Int, y : Int) \\Rightarrow Int[\\uparrow x, \\uparrow y] \\)

\n\n

2.) minus : \\( (x : Int, y : Int) \\Rightarrow Int[\\uparrow x, \\downarrow y] \\)

\n\n

An sfun type, written with \\(\\Rightarrow\\) rather than \\(\\rightarrow\\), names its formal arguments and also qualifies each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \\(\\uparrow\\) is associated with arguments that are separately monotone and \\(\\downarrow\\) is associated with arguments that are separately antitone. The second argument of a binary function \\(f\\) is separately antitone if \\(p_2 \\leq p_2'\\) implies \\(f(p_1, p_2) \\geq f(p_1, p_2')\\).

\n\n

Terms outside of sfun abstractions are typed using a global typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.

\n\n

\\( \\Gamma \\vdash t : T \\)

\n\n

A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:

\n\n

\\( \\Gamma;\\Omega;\\Phi \\vdash t : T \\)

\n\n

Here the global type environment \\( \\Gamma \\) contains all of the variables bound outside of the sfun, the ambient type environment \\( \\Omega \\) contains the list of the sfun’s formal arguments, and the lifted type environment \\( \\Phi \\) contains those variables in \\( t \\)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let’s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.

\n\n

$$\\frac{\\Gamma;x:Int;x:Int[=~x] \\vdash plus(x,x) : Int[\\uparrow~x]} {\\Gamma \\vdash \\tilde{\\lambda} x : Int. plus(x,x) : ( x : Int ) \\Rightarrow Int[\\uparrow~x]}$$

\n\n

Here we type a single-argument sfun abstraction \\(\\tilde{\\lambda} x:Int. plus(x,x)\\). As you might have guessed, \\(\\tilde{\\lambda}\\) is used rather that \\(\\lambda\\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction’s bound variable \\(x\\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \\(x\\) in a type (for example \\(Int[\\uparrow x]\\)) does not refer to some integer, but instead a “slot” named \\(x\\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction’s formal argument \\(x\\), and therefore we add \\(x : Int\\) to the ambient environment. We would also like to include occurrences of \\(x\\) as terms in the body of the abstraction; for these, we add the entry \\(x : Int[=~x]\\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \\(x\\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \\(x\\) in Int[\\(\\uparrow x\\)] refers to the entry for \\(x\\) in the ambient type environment rather than the one in the lifted type environment.

\n\n

The premise of the above rule application includes the strange looking types \\(Int[=~x]\\) and \\(Int[\\uparrow~x]\\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \\(Int\\), and we would expect our abstraction’s body \\(plus(x,x)\\) to have type \\(Int\\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function after it has been applied. Unfortunately, this isn’t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, before it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \\(x\\) in the term \\(plus(x,x)\\) has type \\(Int[=~x]\\), meaning that it is a function which takes the value provided to \\(x\\) (the enclosing sfun’s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called ambient maps. The type \\(Int[=~x]\\) happens to be a singleton type, containing the set of pairs \\(\\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \\ldots \\}\\).

\n\n

The sfun application \\(plus(x,x)\\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \\(x\\) are forwarded into the left and right arguments of the sfun \\(plus\\). The domain of this composite function matches the domain \\(x:Int\\) of the enclosing sfun, which it inherits from the two occurrences of \\(x\\). Since \\(plus\\) returns an \\(Int\\), so does the composite function \\(plus(x,x)\\). The premise of the above typing rule application tells us that \\(plus(x,x)\\) has type \\(Int[\\uparrow~x]\\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise’s derivation, but a full treatment is outside the scope of this post.

\n\n

Since lifted typing is all about function composition, one might wonder how we treat occurrences of \\( \\Gamma \\)’s variables within the body of an sfun. Such a variable might have the type \\( Int \\), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \\( Int \\), for example, to occur in a context where terms of type \\( Int[ \\uparrow x ] \\) are expected. A constant function \\(f\\), after all, is monotone: \\( v_1 \\leq v_2 \\) implies \\( f(v_1) = c \\leq c = f(v_2) \\).

\n\n

We’re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \\( \\Gamma \\vdash t : T \\) is derivable, the execution of the term \\(t\\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \\( \\Gamma \\vdash t : T \\) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \\( t \\) is guaranteed to terminate, producing a value of type \\(T\\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or “computation model”) which can be viewed in one of two ways:

\n\n
    \n
  1. The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.
    2. \n
  2. The composition of an sfun’s components, before the sfun is ever applied.
\n\n

Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications—specifically two applications in the case of monotonicity. If the lifted reduction of an sfun’s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun’s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \\( \\cdot;x:Int;x:Int[=~x] \\vdash plus(x,x) : Int[\\uparrow~x] \\) implies that \\( plus(\\{ (0,0), (1,1), \\ldots \\},\\{ (0,0), (1,1), \\ldots \\} ) \\) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \\( \\tilde{\\lambda} x : Int. plus(x,x) \\) behaves monotonically under terminal reduction. It’s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.

\n\n

Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \\( f(g(v)) \\) and \\( (f \\circ g) (v) \\). The expression \\( f(g(v)) \\) presents function composition as viewed by a standard type systems: to apply the composition of \\(f\\) and \\(g\\) to a value \\(v\\), we first apply \\(g\\) to \\(v\\), and then apply \\(f\\) to the result. This isn’t wrong, but if \\( f \\) and \\( g \\) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the “lifted reduction step” \\( f(g(v)) \\to (f \\circ g) (v) \\).

\n\n

We’ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn’t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:

\n\n
    \n
  1. X increments, resulting in a count of 1
    2. \n
  2. X sends a synchronization message to Y, containing X’s count 1
    3. \n
  3. X receives a synchronization message from Y containing a count of 1
\n\n

At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map’s finite representation maps to 0. Such counters are called GCounters, and described in detail by [Shapiro et al. (2011)].

\n\n

GCounters are partially ordered componentwise. We write \\( v[a] \\) for the natural number to which the GCounter \\(v\\) maps the replica identifier \\(a\\), and we write \\( \\leq \\) for the standard ordering on natural numbers. The partial order \\( \\leq' \\) on GCounters is then defined such that \\( v \\leq' w \\) whenever for all replica identifiers \\(a\\) we have \\( v[a] \\leq w[a] \\).

\n\n

[Meiklejohn et al. (2015)] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and > are standard arithmetic operators for working with natural numbers.

\n\n
getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k]\njoinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n]\nspan :: (m:NatMap) ⇒ Nat[↑ m]\nmax :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b]\nemptyMap :: NatMap\n+ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y]\n> :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y]\n\ntype GCounter = { map : NatMap }\n\nsfun sumCounters(x : GCounter, y : GCounter) \n : GCounter[↑ x, ↑ y] =\n let xMap : NatMap[↑ x, ↑ y] = x.map\n let yMap : NatMap[↑ x, ↑ y] = y.map\n let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap)\n fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) \n  : NatMap[↑ x, ↑ y] =\n  let cond : Bool[↑ x, ↓ y] = k > maxSpan\n   if cond then\n    acc\n   else\n    let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k))\n    sumCell (k+1) acc'\n let initMap : IntArray[↑ x, ↑ y] = emptyMap\n GCounter { map = sumCell 0 initMap }
\n\n

While our system can handle much of this example, it can’t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [Vazou et al. (2014)]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.

\n\n

So there’s still much to do! If you’re interested in more details behind the type system, have a look at Kevin’s blog article, Monotonicity Through Types, or have a look at the full Monotonicity Types preprint for more.

\n\n

References

\n\n

C. Meiklejohn and P. Van Roy. Lasp: A language for distributed, coordination-free programming. In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.

\n\n

N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. Logic and lattices for distributed programming. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.

\n\n

M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. Conflict-Free replicated data types. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.

\n\n

L. Kuper and R. R. Newton. LVars: Lattice-based data structures for deterministic parallelism. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.

\n\n

N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. Refinement types for Haskell. SIGPLAN Not. 49, 9 (August 2014), 269–282.

")) ((? . 85) f post (u . "Type-Directed Compilation, Parts I and II") (? . 85) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/index.html" . unix) (u . "/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/") (u . "2017-04-17T12:00:17") (? . 15) (? . 84) (c (u . "HOPL") c (u . "Author: Leif Andersen") c (u . "Author: William J. Bowman")) (? . 5) #t (u . "\n\n

Part I: Type-Directed Compilation, by Leif Andersen.

\n\n

In this talk we discuss the history of type directed compilation. We start with Xavier Leroy’s seminal paper: Unboxed Objects and Polymorphic Typing, continue to TIL (Typed Intermediate Language), and finish up with TAL (Typed Assembly Language). We talk about what it means for a compiler to be typed preserving, and give examples of optimizations that are enabled by types.

\n\n

Discussion summary:

\n\n\n\n

Part II: Dependent Type-Directed Compilation, by William J. Bowman

\n\n

A certifying compiler is not verified, but it produces a proof of correctness for each binary. This proof can be independently checked to show that the binary was compiled correctly, removing the compiler from the trusted code base. Certifying compilation has its roots in preserving type-preserving compilation, and in particular in preserving dependent types. We start the history of dependent-type-preserving compilation with a compiler from C to Assembly. We’ll see a result showing that preserving dependent types isn’t possible, and then we’ll do it anyway.

\n\n

Discussion summary:

\n\n\n\n

Notes (to appear here, eventually):

\n\n")) ((? . 86) f post (u . "PRL at SNAPL'17") (? . 86) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/04/25/prl-at-snapl-17/index.html" . unix) (u . "/blog/2017/04/25/prl-at-snapl-17/") (u . "2017-04-25T16:46:54") (? . 84) (? . 87) (c (u . "Author: Gabriel Scherer")) (u . "\n

PRL recently produced three papers for the SNAPL conference.

\n\n") #t (u . "\n

PRL recently produced three papers for the SNAPL conference.

\n\n\n\n\n

Linking Types for Multi-Language Software: Have Your Cake and Eat It Too

\n\n

Daniel Patterson and Amal Ahmed, 2017

\n\n
\n

Software developers compose systems from components written in many different languages. A business-logic component may be written in Java or OCaml, a resource-intensive component in C or Rust, and a high-assurance component in Coq. In this multi-language world, program execution sends values from one linguistic context to another. This boundary-crossing exposes values to contexts with unforeseen behavior—that is, behavior that could not arise in the source language of the value. For example, a Rust function may end up being applied in an ML context that violates the memory usage policy enforced by Rust’s type system. This leads to the question of how developers ought to reason about code in such a multi-language world where behavior inexpressible in one language is easily realized in another.

\n

This paper proposes the novel idea of linking types to address the problem of reasoning about single-language components in a multi-lingual setting. Specifically, linking types allow programmers to annotate where in a program they can link with components inexpressible in their unadulterated language. This enables developers to reason about (behavioral) equality using only their own language and the annotations, even though their code may be linked with code written in a language with more expressive power.

\n\n

Search for Program Structure

\n\n

Gabriel Scherer, 2017.

\n\n
\n

The community of programming language research loves the Curry-Howard correspondence between proofs and programs. Cut-elimination as computation, theorems for free, ‘call/cc’ as excluded middle, dependently typed languages as proof assistants, etc.

\n

Yet we have, for all these years, missed an obvious observation: “the structure of programs corresponds to the structure of proof search”. For pure programs and intuitionistic logic, more is known about the latter than the former. We think we know what programs are, but logicians know better!

\n

To motivate the study of proof search for program structure, we retrace recent research on applying the logical technique of focusing to study the canonical structure of simply-typed λ-terms. We then motivate the open problem of extending canonical forms to support richer type systems, such as polymorphism, by discussing a few enticing applications of more canonical program representations.

\n\n

Migratory Typing: Ten Years Later

\n\n

Sam Tobin-Hochstadt, Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Ben Greenman, Andrew M. Kent, Vincent St-Amour, T. Stephen Strickland and Asumu Takikawa, 2017.

\n\n
\n

In this day and age, many developers work on large, untyped code repositories. Even if they are the creators of the code, they notice that they have to figure out the equivalent of method signatures every time they work on old code. This step is time consuming and error prone.

\n

Ten years ago, the two lead authors outlined a linguistic solution to this problem. Specifically they proposed the creation of typed twins for untyped programming languages so that developers could migrate scripts from the untyped world to a typed one in an incremental manner. Their programmatic paper also spelled out three guiding design principles concerning the acceptance of grown idioms, the soundness of mixed-typed programs, and the units of migration.

\n

This paper revisits this idea of a migratory type system as implemented for Racket. It explains how the design principles have been used to produce the Typed Racket twin and presents an assessment of the project’s status, highlighting successes and failures.

\n\n

.

\n\n

SNAPL is not dissimilar to the (french-speaking) JFLA that I am more familiar with — with an added irritating call for paper and unreasonable registration price. It has an interesting diversity of topics of presentation: see also the complete list of accepted papers this year, and the list of the previous edition.

")) ((? . 32) f post (u . "[Conversational Concurrency (cross-post)](https://eighty-twenty.org/2018/01/24/conversational-concurrency)") (? . 32) 1731622765 (p+ #"/home/runner/work/website/website/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/index.html" . unix) (u . "/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/") (u . "2019-05-11T00:03:16") (? . 31) (? . 11) (c (u . "dissertation") c (u . "Author: Tony Garnock-Jones")) (? . 5) #f (? . 5)) ((? . 87) f post (u . "What is Soft Typing?") (? . 87) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/04/28/what-is-soft-typing/index.html" . unix) (u . "/blog/2017/04/28/what-is-soft-typing/") (u . "2017-04-28T12:25:17") (? . 86) (? . 12) (c (u . "HOPL") c (u . "Author: Ben Greenman")) (? . 5) #t (u . "\n\n

A soft type system rewrites programs and meets a few design criteria.

\n\n
\n\n

What are the Design Criteria?

\n\n

According to Mike Fagan’s 1991 dissertation, a soft type system must:

\n\n\n\n

Important details:

\n\n\n\n

Why would I want to use a soft type system?

\n\n

If you:

\n\n\n\n

then Soft Typing is a perfect fit. You just need to find/build a soft type checker.

\n\n

Clarification

\n\n

The benefits of static typing that a soft type system can give are:

\n\n\n\n

See Andrew Wright’s 1994 dissertation for proof.

\n\n

Can I use Andrew Wright’s soft type system?

\n\n

Not sure, but you may download the code for it:

\n\n\n\n

Please explain Fagan’s / Wright’s soft types

\n\n

Types t are made of constructors k, flags f, and type variables a. The grammar for types is basically:

\n\n
  t ::= a | (k f t ...) U t\n  k ::= Int | Pair | ->\n  f ::= ++ | -- | b\n  a ::= a0 | a1 | a2 | a3 | ....\n  b ::= b0 | b1 | b2 | b3 | ....
\n\n

where:

\n\n\n\n

There are also some rules for types to be well-formed.

\n\n

Here are two well-formed types:

\n\n
(Int ++) U a0\n\n(-> ++ ((Int b0) U a1)\n       ((Int ++) U a2)) U a3
\n\n

Here are two types that match the grammar, but are NOT well-formed:

\n\n
(Int ++ a0) U a1\n\n(-> --) U a2
\n\n

Finally, some intuition:

\n\n\n\n

The type and flag variables let Fagan and Wright encode subtyping using polymorphism. It’s a very cool idea, introduced in Didier Remy’s POPL 1989 paper. But it adds a learning curve, and has some drawbacks for type inference.

\n\n

Stream-of-consciousness notes from the HOPL lecture

\n\n")) ((? . 23) f post (u . "No Good Answers, Gradually Typed Object-Oriented Languages") (? . 23) 1731622765 (p+ #"/home/runner/work/website/website/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/index.html" . unix) (u . "/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/") (u . "2017-05-09T14:04:31") (? . 3) (? . 22) (c (u . "HOPL") c (u . "Gradual Typing") c (u . "Author: Ben Chung")) (? . 5) #t (u . "\n\n

Untyped code remains a real problem in practice, as a result of reduced performance and hindered readability. One approach to solve this problem is gradual typing.

\n\n

Gradual typing puts the onus on the developer to add type annotations, statically checks whatever type annotations have been written, and dynamically ensures that untyped code does not violate those annotations. A number of approaches have been put forward to try to achieve these objectives while retaining efficiency, semantic meaning, and the ability to actually type untyped code.

\n\n

I discussed three systems, all of which achieve the objective of typing untyped code in different ways, and all of which have different tradeoffs.

\n\n

Unofficial Notes:

\n\n\n\n

Code Examples:

\n\n")) ((? . 57) f post (u . "Meaningful Distinctions") (? . 57) 1731622765 (p+ #"/home/runner/work/website/website/blog/2016/10/31/meaningful-distinctions/index.html" . unix) (u . "/blog/2016/10/31/meaningful-distinctions/") (u . "2016-10-31T17:20:33") (? . 55) (? . 13) (c (u . "history") c (u . "constructions") c (u . "Author: Ben Greenman")) (u . "\n
\n

“Meaningful distinctions deserve to be maintained.” — Errett A. Bishop

\n\n

Likewise, memorable quotations deserve to be read in context. In this spirit, I am happy to present the above “basic principle” in its context: Schizophrenia in contemporary mathematics (pdf)

\n\n

Read on for a brief summary.

") #t (u . "\n
\n

“Meaningful distinctions deserve to be maintained.” — Errett A. Bishop

\n\n

Likewise, memorable quotations deserve to be read in context. In this spirit, I am happy to present the above “basic principle” in its context: Schizophrenia in contemporary mathematics (pdf)

\n\n

Read on for a brief summary.

\n\n\n
\n\n

I first read the above quotation in Michael Beeson’s introduction to the 2012 edition of Bishop’s Foundations of Constructive Analysis. That was two years ago.

\n\n

Last month, I tried to find its context. Many other uses of the quote cited a Schizophrenia in comtemporary mathematics, but I could not find an electronic copy. (It turns out, the AMS Bookstore page for Erret Bishop: Reflections on Him and His Research includes a facsimile.)

\n\n

Lest anyone else be tempted to conjure the ancient magic of inter-library loan, here is a scan of the pages I borrowed. Thanks to the University of Toledo for supplying the hard copy.

\n\n
\n

prl.ccs.neu.edu/img/sicm.pdf

\n\n

The document is Bishop’s “feeling for the philosophical issues involved” in constructive mathematics. First, Bishop lists “schizophrenic attributes” (trouble spots) of contemporary mathematics. Next, he gives basic principles of constructivism and Brouwer’s interpretation of the logical quantifiers. Along the way, and as a source of examples, Bishop describes integers, sets, and real numbers. The emphasis is always on common-sense meaning and finite constructions.

\n\n

After a brief summary and reflection, the last ten pages list recent advances in constructive mathematics and upcoming tasks. The open tasks are particularly interesting:

\n\n\n\n

The popular quote on “Meaningful Distinctions” appears early in the paper, as one of Bishop’s four principles that “stand out as basic” to the philosophy of constructivism:

\n\n
\n

A. Mathematics is common sense.

\n

B. Do not ask whether a statement is true until you know what it means.

\n

C. A proof is any completely convincing argument.

\n

D. Meaningful distinctions deserve to be maintained.

\n\n

I had no idea that D was “a principle”, or that it had three siblings.

\n\n

To further tempt you into reading the whole truth, here are some of my favorite phrases:

\n\n
\n
\n\n

Now go find their context!

")))) \ No newline at end of file diff --git a/blog/2016/04/29/welcome-to-the-prl-blog/index.html b/blog/2016/04/29/welcome-to-the-prl-blog/index.html new file mode 100644 index 00000000..53c060be --- /dev/null +++ b/blog/2016/04/29/welcome-to-the-prl-blog/index.html @@ -0,0 +1,190 @@ + + + + + + Welcome to the PRL blog + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Welcome to the PRL blog

+

+ :: about, 1st blog post

+

By: Ben Greenman

+
+ +

Greetings, ground rules, hopes, dreams, and notes for contributors. Welcome aboard.

+ + +

Earlier this year, the Programming Research Lab (PRL) was blessed with a new postdoc: Gabriel Scherer from INRIA Paris-Rocquencourt, France. Ever since Gabriel arrived things have been changing here in Boston. We now have homemade bread on the first Tuesday of every month, orange water crepes after holidays, and someone new to go out for bubble tea with in between. All that and an enthusiastic colleague and researcher.

+ +

In his spare time between lobbying the CS department for an espresso machine and building multi-language compilers, Gabriel is also a champion of open access. Hence this blog, a window into the life and times of PRL students made possible by Gabriel’s tactical prodding and careful delegation of responsibilities. Anything you might read about in a rejected conference paper or hear over coffee is fair game here: the goal is to give the wide world a glimpse of our lab and people.

+ +

For Contributors

+ +

These pages are generated using Greg Hendershott’s frog static website generator. To create a new post:

+ +
    +
  1. Clone or fork the nuprl.github.io repository
    2. +
  2. Check out a new git branch for your post
    3. +
  3. Run cd blog; raco frog -n "TITLE" to build a template for a new post
    4. +
  4. Add content to the new markdown file (under _src/posts)
    5. +
  5. Rebuild the blog with raco frog -b
    6. +
  6. Run cd ..; raco frog -p to start a web server and view your changes at http://localhost:3000/
    7. +
  7. Send a pull request to the nuprl.github.io repo
+ +

An open pull request is the best place to ask questions about the formatting or content of a post. We promise that within a few days of opening a PR someone with push access will reply with feedback or merge the request.

+ +

Contributions are open to anyone: current labmates, alumni, friends from the Racket mailing list, and even recovering C programmers. One should have a strong connection to Northeastern or our research, but even that is not strictly necessary. Visitors are always welcome to the PRL.

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/index.html b/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/index.html new file mode 100644 index 00000000..2a743721 --- /dev/null +++ b/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/index.html @@ -0,0 +1,183 @@ + + + + + + NEPLS on May 31st at UMass, Amherst + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

NEPLS on May 31st at UMass, Amherst

+

+ :: NEPLS, conference

+

By: Gabriel Scherer

+
+ +

It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!

+ + +
+

The next New England Programming Languages and Systems Symposium will take place on Tuesday, May 31st 2016 at University of Massachusetts, Amherst. Please mark it in your calendars!

+

The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.

+

Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.

+

The submission deadline is Tuesday, May 17th. Send your proposal to talks@nepls.org.

+

More details about NEPLS are available on the NEPLS webpage:

+

http://www.nepls.org/

+ +

I’m personally fond of such regional events, which is a great time to learn about the research around us in a less formal and exhausting setting than a 200-attendees conference.

+ +

If you are in the area, please consider applying to talk about your work. If one of your colleague is working on something you find exciting, please invite them to apply!

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/05/18/gradual-typing-across-the-spectrum/index.html b/blog/2016/05/18/gradual-typing-across-the-spectrum/index.html new file mode 100644 index 00000000..f4a735d4 --- /dev/null +++ b/blog/2016/05/18/gradual-typing-across-the-spectrum/index.html @@ -0,0 +1,237 @@ + + + + + + Gradual Typing Across the Spectrum + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Gradual Typing Across the Spectrum

+

+ :: gradual typing, PI meeting

+

By: Asumu Takikawa

+
+ +
+

Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. — Matthias Felleisen

+ +

Yesterday we hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found here). In case you aren’t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.

+ +

Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing—the ideas that cross language and platform barriers.

+ + +

There were a good number of both institutions and programming languages represented at the meeting, with researchers from all of Brown University, Indiana University, Northeastern University, and the University of Maryland. The languages that we work on cover a broad subset of the dynamically-typed languages: Clojure, JavaScript, R, Racket, Ruby, Pyret, and Python.

+ +
+

+ +

The specific research artifacts that were represented include Reticulated Python, RDL (contracts for Ruby), StrongScript, Typed Clojure, and Typed Racket.

+ +

In this blog post, I’ll summarize some of the key research themes that were brought up at the meeting. Since I can’t go into too much detail about every topic, I will link to the relevant research papers and other resources.

+ +

At a high level, the talks covered four major facets of gradual typing: expressiveness, performance, usability, and implementation techniques.

+ +

Expressiveness

+ +

By expressiveness, I mean what kinds of language features a gradual type system supports and the richness of the reasoning that the type system provides. Since gradual typing is about augmenting existing dynamically-typed languages, a gradual type system should support the language features that programmers actually use.

+ +

This is why recent implementations of gradual typing have focused on enabling object-oriented programming, since objects are widely used in nearly all dynamically-typed languages in use today. Unfortunately, since different languages have wildly different object systems, it’s hard to compare research on gradual OO languages. Ben Chung is working to address this by coming up with a formal model that tries to unify various accounts of objects in order to better explain the design tradeoffs. His goal is to cover the core ideas in Reticulated Python, StrongScript, and Typed Racket.

+ +

Of course, dynamically-typed languages have a lot more than OO features. Along these lines, I gave a talk on how at NU we’re working to extend Typed Racket to cover everything from objects (my thesis topic), first-class modules (Dan Feltey’s MS project), and higher-order contracts (Brian LaChance’s MS project).

+ +

On the other side, as programs get more complex, programmers may wish to write richer type specifications that provide even more guarantees. This makes gradual typing a wide spectrum that goes from completely untyped, fully typed, and then beyond to dependently typed. Andrew Kent and David Christiansen both presented work that takes gradual typing beyond ordinary typed reasoning with dependent types.

+ +

Andrew presented an extension of Typed Racket that adds type refinements that can check rich properties (like vector bounds) that are found in real Racket code (see his RacketCon talk and recent PLDI paper). David Christiansen followed with a talk about adding dependent type theory to Typed Racket, which would allow correct-by-construction programming using a Nuprl-like proof system (he had a very cool GUI proof assistant demo in his slides!).

+ +

Performance

+ +
+

+ +

One of the key practical concerns about gradual typing is its performance overhead. It’s a concern because in order to ensure type safety, a gradually-typed language implementation needs to install dynamic checks between the typed and untyped parts of a program. This catches any inconsistencies between the typed interfaces and how the untyped code may call into them.

+ +

Ben Greenman gave an upbeat talk that set the stage for this topic, pointing out some key lessons that we’ve learned about performance from building Typed Racket. The main idea he presented (also the topic of our POPL 2016 paper) is that to evaluate a gradual type system, you want to explore different ways of adding types to a program and see how much it costs. This evaluation effort started with Typed Racket, but he and Zeina Migeed are working on expanding it to Reticulated Python.

+ +

From IU, Andre Kuhlenschmidt and Deyaaeldeen Almahallawi are exploring how ahead-of-time (AOT) compilation strategies could help reduce the cost of gradual typing. In particular, they are working on implementing Schml: a compiler from the gradually-typed lambda calculus to C.

+ +

In addition to AOT compilation, the folks at IU are exploring tracing JIT compilation as a means to make gradual typing faster. More specifically, Spenser Bauman talked about Pycket, an alternative implementation of Racket that uses RPython/PyPy to dramatically lower the overhead of gradual typing (also see the recording of Spenser’s talk on the topic at RacketCon and his ICFP paper).

+ +

Usability

+ +

On the usability side, both Shriram Krishnamurthi and Ambrose Bonnaire-Sergeant made observations on what it takes to get gradual typing in the hands of real software developers.

+ +

Shriram approached the topic from the angle of CS education, which is the focus of the Pyret language, and shared what the Brown language group is working on. While Pyret doesn’t exactly fit the mold of gradual typing, it’s a close cousin since it’s a dynamically-typed language that explicitly takes design cues from the best parts of statically-typed languages. That approach lets CS beginners think in terms of types (the approach spearheaded by HtDP and Bootstrap) without having to battle a typechecker from the start.

+ +

For professional software developers, a major concern with gradual typing is that writing type annotations may be a tedious and time intensive task. Ambrose, who is the creator of Typed Clojure, shared some preliminary work on how to cut down on the tedium by inferring gradual type annotations by instrumenting programs for a dynamic analysis. The hope is to be able to infer both recursive and polymorphic type annotations automatically from tests (you may also be interested in Ambrose’s recent ESOP paper on Typed Clojure).

+ +

Implementation Techniques

+ +

Finally, several talks focused on alternative implementation techniques for gradual typing that provide a variety of software engineering benefits for implementers.

+ +

From Maryland, Brianna Ren gave a talk on Hummingbird, a just-in-time typechecker for Ruby programs (also see the upcoming PLDI paper by Brianna and Jeff Foster). The basic idea is that it’s hard to implement a traditional static typechecker for a language that heavily relies on metaprogramming, in which the fields/methods of classes may be rewritten at run-time. This is particularly tricky for frameworks like Ruby on Rails. Instead of checking types at compile-time, Hummingbird actually executes the typechecker at run-time in order to be able to accurately check programs that use run-time metaprogramming. To reduce overheads, she uses a cache for typechecking that is invalidated when classes are modified.

+ +

Stephen Chang gave a very different view on metaprogramming in his talk, which focused on implementing typecheckers using metaprogramming (the trivial Typed Racket library is an offshoot of this work). His key idea is that typecheckers share many aspects with macro-based metaprogramming systems, such as the need to traverse syntax and annotate it with information. Since they share so much in common, why not just implement the typechecker as a macro? Stephen demonstrates that not only is this possible, but that it’s possible to implement a wide variety of type system features this way including (local) type inference. The connection to gradual typing is that even a gradual type system can be implemented as a metaprogram by integrating the generation of dynamic checks into the macro transformation process.

+ +

The last talk of the day (but certainly not the least), was by Michael Vitousek, who focused on the transient implementation of gradual typing (first described in his DLS paper). Traditionally, gradual type systems have implemented their dynamic checks using proxy objects that wrap method implementations with both pre- and post-checks. Unfortunately, this implementation technique often conflicts with the underlying language. Since proxying changes the identity of an object it can interfere with object equality tests. Instead, the transient approach bakes the dynamic checks into and throughout the typed code to implement a “defense in depth” against inconsistencies with untyped code. The great thing about this implementation technique is that it doesn’t demand any specialized support from the underlying language runtime and is therefore easy to port to other languages (like JavaScript).

+ +

Conclusion

+ +
+

+ +

Hopefully this blog post helps provide a better picture of the state of gradual typing research. The exciting thing about gradual typing is that it contains both interesting theoretical problems and also connects to the practical needs of software developers.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/index.html b/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/index.html new file mode 100644 index 00000000..b2cf1ad3 --- /dev/null +++ b/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/index.html @@ -0,0 +1,609 @@ + + + + + + Measuring GC latencies in Haskell, OCaml, Racket + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + + +

The Haskell benchmark

+ +

James Fisher’s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly — a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
 +
+ 
import qualified Control.Exception as Exception
+import qualified Control.Monad as Monad
+import qualified Data.ByteString as ByteString
+import qualified Data.Map.Strict as Map
+
+data Msg = Msg !Int !ByteString.ByteString
+
+type Chan = Map.Map Int ByteString.ByteString
+
+message :: Int -> Msg
+message n = Msg n (ByteString.replicate 1024 (fromIntegral n))
+
+pushMsg :: Chan -> Msg -> IO Chan
+pushMsg chan (Msg msgId msgContent) =
+  Exception.evaluate $
+    let inserted = Map.insert msgId msgContent chan in
+      if 200000 < Map.size inserted
+      then Map.deleteMin inserted
+      else inserted
+
+main :: IO ()
+main = Monad.foldM_ pushMsg Map.empty (map message [1..1000000])
+
+
+
+ +

To compile and run the program (make run-haskell also works in my repository):

+ +
ghc -O2 -optc-O3 Main.hs  # compile the program
+./Main +RTS -s            # run the program (with GC instrumentation enabled)
+ +

On my machine, running the program takes around 1.5s. We are not interested in the total running time (the throughput of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post — and there it is considered excessive, with an expected worst-case latency of at most “a few milliseconds”.

+ +

(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)

+ +

This Haskell code makes two assumption about the Map data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus Map.size is constant-time — for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key — OCaml does this as well, but Racket uses hashes instead.

+ +

I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it’s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
 +
+ 
type Msg = ByteString.ByteString
+type Chan = Map.Map Int Msg
+
+windowSize = 200000
+msgCount = 1000000
+
+message :: Int -> Msg
+message n = ByteString.replicate 1024 (fromIntegral n)
+
+pushMsg :: Chan -> Int -> IO Chan
+pushMsg chan highId =
+  Exception.evaluate $
+    let lowId = highId - windowSize in
+    let inserted = Map.insert highId (message highId) chan in
+    if lowId < 0 then inserted
+    else Map.delete lowId inserted
+
+main :: IO ()
+main = Monad.foldM_ pushMsg Map.empty [0..msgCount]
+
+
+
+ +

This variant has the same running times and worst-case pause, 50ms, as the original program.

+ +

Explaining Haskell results

+ +

James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC’s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.

+ +

The original blog post contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.

+ +

As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell’s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.

+ +

OCaml version

+ +

The code can easily be ported into OCaml, for example as follows:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
 +
+ 
open Batteries
+module IMap = Map.Make(Int)
+
+let message n = String.make 1024 (Char.chr (n mod 256))
+
+let window_size = 200_000
+let msg_count = 1_000_000
+
+let push_msg chan high_id =
+  let low_id = high_id - window_size in
+  let inserted = IMap.add high_id (message high_id) chan in
+  if low_id < 0 then inserted
+  else IMap.remove low_id inserted
+
+let () =
+  Seq.init msg_count (fun i -> i)
+  |> Seq.fold_left push_msg IMap.empty |> ignore
+
+
+
+ +

Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.

+ +

To measure the maximal pause time, there are two options:

+ +
    +
  • +

    use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it’s far from being as convenient as GHC’s +RTS -s parameter.

    • +
  • +

    Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, make run-ocaml)

+ +

To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the --with-instrumented-runtime configure-time switch. Then, you can use the i-variant (i for “instrumented”) of the runtime that is compiled with instrumentation enabled. (My makefile rule make +run-ocaml-instrumented does this for you, but you still need a switch compiled with the instrumented runtime.)

+ +
ocamlbuild -tag "runtime_variant(i)" main.native
+OCAML_INSTR_LOG=ocaml.log ./main.native
+ +

The log file ocaml.log will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools ocaml-instr-report and ocaml-instr-graph of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is dispatch, which contains the time spent in all GC activity. The relevant section of ocaml-instr-report’s output shows:

+ +
==== dispatch: 2506
+470ns..1.0us:  1     (768ns)                       0.04%
+1.0us..2.2us: # 2                                  0.12%
+2.2us..4.7us: ### 8                                0.44%
+4.7us..10us : #### 10                              0.84%
+ 10us..22us :  1     (14us)                        0.88%
+ 22us..47us :                                      0.88%
+ 47us..100us:                                      0.88%
+100us..220us: ## 3                                 1.00%
+220us..470us: ########## 668                      27.65%
+470us..1.0ms: ########### 1795                    99.28%
+1.0ms..2.2ms: ##### 17                            99.96%
+2.2ms..4.7ms:  1     (2.7ms)                     100.00%
+ +

As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.

+ +

The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration — as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
 +
+ 
let worst = ref 0.
+let time f =
+  let before = Unix.gettimeofday () in
+  let result = f () in
+  let after = Unix.gettimeofday () in
+  worst := max !worst (after -. before);
+  result
+
+let push_msg chan high_id = time @@ fun () ->
+  let low_id = high_id - window_size in
+  let inserted = IMap.add high_id (message high_id) chan in
+  if low_id < 0 then inserted
+  else IMap.remove low_id inserted
+
+(* ..main loop.. *)
+let () = Printf.printf "Worst pause: %.2E\n" !worst
+
+
+
+ +

Running this version reports a worst-case latency of 2ms seconds on my machine (I use the %E formatter for scientific notation, so it gets printed as 2.03E-03), which is in line with the instrumented runtime — actually slightly lower, as the instrumentation may add some overhead.

+ +

A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.

+ +

Explaining OCaml results

+ +

The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.

+ +

The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency instrumentation tools that I’m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.

+ +

Racket version

+ +

Max New wrote a first version of Racket port of this benchmark — he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
 +
+ 
#lang racket/base
+(require racket/match)
+
+(define window-size 200000)
+(define msg-count  2000000)
+
+(define (message n) (make-bytes 1024 (modulo n 256)))
+
+(define (push-msg chan id-high)
+  (define id-low (id-high . - . window-size))
+  (define inserted (hash-set chan id-high (message id-high)))
+  (if (id-low . < . 0) inserted
+      (hash-remove inserted id-low)))
+
+(define _
+  (for/fold
+     ([chan (make-immutable-hash)])
+     ([i (in-range msg-count)])
+     (push-msg chan i)))
+
+
+
+ +

I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:

+ +
    +
  • +

    Sam Tobin-Hochstadt’s gcstats package makes Racket programs produce a summary of their runtime behavior in the same format as GHC’s +RTS -s output, with in particular the worst-case pause time. It is also very easy to use:

    + 
    racket -l gcstats -t main.rkt
    • +
  • +

    By setting the environment variable PLTSTDERR=debug@GC, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:

    + 
    cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count
+ +

Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable PLT_INCREMENTAL_GC=1. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.

+ +

On my machine, using the latest release Racket 6.5, the maximal pause time reported by gcstats is around 150ms, which is rather bad — the excessive pause of GHC was 50ms.

+ +

Investigating the Racket results

+ +

I sent an email to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.

+ +

First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.

+ +

Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the “ramp-up” period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran make +run-racket-instrumented. I can look at the pause time of major collections by doing grep MAJ racket.log, and on my machine I have:

+ +
GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373
+GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521
+GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810
+GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571
+GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321
+GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046
+GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553
+GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048
+GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506
+GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983
+GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438
+GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907
+GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362
+GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897
+GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348
+GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827
+GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290
+GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774
+GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251
+GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730
+GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254
+GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730
+GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220
+GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697
+ +

Look at the evolution of major collection pause times: there is an early peek at 140ms, but then pause times decrease and the steady state has sensibly shorter pauses of around 22ms. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.

+ +

My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.

+ +

This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it’s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer’s post, and would not be a problem during the steady state of an actual message-passing application.

+ +

Tuning the Racket version

+ +

Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket’s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.

+ +

I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime — around the iteration count that corresponds to the maximal channel size. I defined a function doing just that

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
 +
+ 
(define (maybe-gc i)
+  (when (and gc-during-rampup
+             (i . > . (window-size . / . 2))
+             (i . < . (window-size . * . 2))
+             (zero? (modulo i 50)))
+        (collect-garbage 'incremental)
+        (collect-garbage 'minor)))
+
+
+
+ +

which is controlled by a gc-during-rampup parameter that you can explicitly set to #t to experiment — explicit GC calls are disabled by default in my benchmark code. Then I just inserted a (maybe-gc i) call in the main loop.

+ +

Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms — the pauses during the steady state are unchanged, around 22ms.

+ +

This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.

+ +

To summarize, Racket’s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/06/07/icfp-2016-looking-for-student-volunteers/index.html b/blog/2016/06/07/icfp-2016-looking-for-student-volunteers/index.html new file mode 100644 index 00000000..affb218e --- /dev/null +++ b/blog/2016/06/07/icfp-2016-looking-for-student-volunteers/index.html @@ -0,0 +1,194 @@ + + + + + + ICFP 2016: looking for student volunteers + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

ICFP 2016: looking for student volunteers

+

+ :: ICFP

+

By: Gabriel Scherer

+
+ +

If you are a student, you should consider applying to become an ICFP 2016 student volunteer! The deadline for application is July 31st, 2016.

+ + +

ICFP 2016, the Internal Conference on Functional Programming, is happening in Nara, Japan. If you are a student, you may be interest in being a Student Volunteer: you help run the conference, and in exchange do not pay registration fees — but you still have to find funding for the travel, hosting, and dinners. Quoting the Student Volunteer webpage:

+ +
+

ICFP is pleased to offer a number of opportunities for student volunteers, who are vital to the efficient operation and continued success of the conference each year. The student volunteer program is a chance for students from around the world to participate in the conferences whilst assisting us in preparing and running the event.

+

Job assignments for student volunteers include assisting with technical sessions, workshops, tutorials and panels, helping the registration desk, operating the information desk, helping with traffic flow, and general assistance to keep the conferences running smoothly.

+

The Student Volunteer Program helps more students attend the ICFP conference by covering conference fees (but not travel or lodging expenses) in exchange for a fixed number of work hours (usually from 8 to 12) helping with the conference organization (registration and information desks, assistance during talk sessions, etc.).

+

The Student Volunteer registration covers:

+
    +
  • Access to all workshops and the main conference,
    • +
  • Daily lunches and coffee breaks,
    • +
  • Access to social events, including the banquet.
+

To apply, please fill the following form.

+

The application deadline is July 31st, 2016. Applications after this date may be considered pending availability.

+

You can send questions about student volunteering to icfp-SV at researchr dot org.

+ +

I was “student volunteer captain” at ICFP last year in Vancouver, and I will do it again this year. My entirely personal take on the thing is that being a Student Volunteer is worth it, but that being a Captain is too much work.

+ +

The main downside of being a volunteer is some of the shifts are at the registration desk, and they may imply missing some of the talks — and also you may have to get up early for your duties. The upsides are many. You get belong to a small group of nice people. You have interactions with many people without much effort; you will enjoy the sparks of gratitude in the eyes of the “Where is Workshop Room B2?” crowd. You have a small peek into the kind of work involved in running a conference; most people actually working on the organization (we SVs are hobbyists) are pouring surprising amount of work. Also, you learn to fold tee-shirts very well, if you’re on “bag stuffing” duty.

+ +

Being a student volunteer can be combined with other forms of travel support, such as SIGPLAN PAC funding; see the travel support page for more details.

+ +

Another thing you should think about is applying to PLMW, the Programming Languages Mentoring Workshop that happens at ICFP, POPL, and PLDI. PLMW funding covers the whole conference cost (travel, housing, registration, dinners), so if you get PLMW funding you have no financial motivation to be a student volunteer. This year, PLMW focuses on early career graduate students.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/index.html b/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/index.html new file mode 100644 index 00000000..d6d5dcf3 --- /dev/null +++ b/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/index.html @@ -0,0 +1,183 @@ + + + + + + Does anyone still care about printed proceedings? (Grab some at NEU this week!) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Does anyone still care about printed proceedings? (Grab some at NEU this week!)

+

+ :: proceedings, dawn of the digital era

+

By: Gabriel Scherer

+
+ +

Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!

+ + +

If you are in the area and are interested, feel free to send me an email and come grab them. We have ASPLOS from XIII to XX, PLDI from 2005 to 2015 (but not 2014), and OOPSLA from 2002 to 2015. When I saw the stack, I grabbed the POPL ones from 2002 to 2015, but in fact I have no idea what to do with them and I’m a bit skeptical they would be of use to me; if you know you would use them, I would be glad to let you have them.

+ +

If you were to buy those proceedings at conference-subscription rates today, it would cost you a small fortune. Yet nobody seems to want them. An odd disconnect, that I found amusing and maybe worthy of a blog post.

+ +

But don’t get me wrong, the future of printed proceedings is not an important question. We should rather be asking ourselves: why are the products of the work of our communities not easily accessible in an Open Access long-term archive? Are you submitting your articles to arXiv, or another archive? Why not?

+ +

Not caring about printed proceedings is perfectly fine; but please care about people outside institutions that want to access your work — for example, master student myself.

+ +
+ +

Update (August 2017): Gabriel returned to France and left the POPL proceedings at his desk. The proceedings are sitting in the hallway at NEU, in case anyone cares.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/06/27/tutorial-using-racket-s-ffi/index.html b/blog/2016/06/27/tutorial-using-racket-s-ffi/index.html new file mode 100644 index 00000000..02330581 --- /dev/null +++ b/blog/2016/06/27/tutorial-using-racket-s-ffi/index.html @@ -0,0 +1,669 @@ + + + + + + Tutorial: Using Racket's FFI + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Tutorial: Using Racket’s FFI

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

Update: this post is now part of a series. Part 2 is +here +and part 3 is +here.

+ +

I’ve seen several people ask for a tutorial on Racket’s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +documentation +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.

+ +

With that in mind, this blog post will provide a step-by-step tutorial +for Racket’s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.

+ + +

Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop paper +that you can read if you’re curious about the design.

+ +

The tutorial will focus on using the Cairo +graphics library, mainly because it comes bundled with Racket.

+ +

To start, let’s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo’s +samples page:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
cairo_move_to (cr, 50.0, 75.0);
cairo_line_to (cr, 200.0, 75.0);
 
cairo_move_to (cr, 50.0, 125.0);
cairo_line_to (cr, 200.0, 125.0);
 
cairo_move_to (cr, 50.0, 175.0);
cairo_line_to (cr, 200.0, 175.0);
 
cairo_set_line_width (cr, 30.0);
cairo_set_line_cap (cr, CAIRO_LINE_CAP_ROUND);
cairo_stroke (cr);
+ +

In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:

+ +
+ + + + + + + +
> (require racket/draw)
> (define bt (make-bitmap 256 256))
> (define bt-surface (send bt get-handle))
+ +

This uses the Racket drawing library racket/draw to construct +a bitmap object that we’ll draw on. The get-handle method just extracts a +low-level Cairo surface value that we can use.

+ +

NB: these code snippets don’t come with a #lang declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy & paste the snippets into your REPL.

+ +

Our first real step is to import the FFI itself:

+ +
+ + + +
> (require ffi/unsafe)
+ +

As the module name suggests, the FFI is unsafe and can cause your Racket process +to segfault. If you’re following along in DrRacket, you will want to save your file +frequently.

+ +

Next, we can load the Cairo library to obtain a foreign-library value, which +is a handle that we use to access C values and functions:

+ +
+ + + +
> (define cairo-lib (ffi-lib #f))
+ +

Since Cairo has already been loaded by the Racket process because of the +racket/gui import earlier, we can supply #f here as an +argument to ffi-lib. Normally you supply the name of a shared +library file such as "libcairo":

+ +
+

(define cairo-lib (ffi-lib "libcairo" '("2" #f)))

+ +

The last list argument specifies the accepted versions (#f allows +a version-less library). For this post, those details aren’t important but +see the docs on ffi-lib if you’re curious.

+ +

Extracting functions

+ +

Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the get-ffi-obj function. The get-ffi-obj +function takes three arguments:

+ +
    +
  • +

    The name of the value as a string (or symbol or bytestring)

    • +
  • +

    a foreign library value, and

    • +
  • +

    a C type, which is a type description that tells +the FFI how to marshall between Racket and C.

+ +

C types are a crucial concept for the FFI. They range from relatively +simple types like _int and _pointer to more complicated +type constructors such as _enum and _fun. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling make-ctype with two functions that +handle marshalling between C and Racket code.

+ +

To make progress with our Cairo code, we need to create a drawing context from +the surface object bt-surface that we defined a while ago. The +relevant function in the Cairo docs is +cairo_create, +which has the following type signature:

+ +
+ + + + + +
/* NB: this is C code */
cairo_t * cairo_create (cairo_surface_t *target);
+ +

+ +
To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a cairo_surface_t and +returns a pointer to a cairo_t. Let’s start with a very simple C type +that matches up with this behavior:
+ +
+
+

(_fun _pointer -> _pointer)

+ +
This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its _fun type.
+ +

The following definition shows how to use this type to obtain a foreign +function:

+ +
+ + + +
+ + + + + + + +
> (define cairo-create
    (get-ffi-obj "cairo_create" cairo-lib
                 (_fun _pointer -> _pointer)))
+ +

Then we can use cairo-create as an ordinary racket function:

+ +
+ + + + + + + +
> (define ctx (cairo-create bt-surface))
> ctx
+

#<cpointer>

+ +

Interlude: more type safety

+ +

Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified _pointer types, it is easy +to accidentally misuse the function:

+ +
+ + + + + + + +
; You may not want to actually run this
; a cairo_t is not a cairo_surface_t
(cairo-create (cairo-create bt-surface))
+ +

To prevent such bad uses, it is good practice to use tagged pointer types +using define-cpointer-type. Here are two example definitions that +correspond to the cairo_t and cairo_surface_t types from earlier:

+ +
+ + + + + + + +
; The leading underscores are mandatory
> (define-cpointer-type _cairo_t)
> (define-cpointer-type _cairo_surface_t)
+ +

We can then redefine cairo-create with a better type, which will +prevent ill-typed calls:

+ +
+ + + + + + + + + + + +
+ + + + + + + +
> (define cairo-create
    (get-ffi-obj "cairo_create" cairo-lib
                 (_fun _cairo_surface_t -> _cairo_t)))
> (cairo-create (cairo-create bt-surface))
+

cairo_surface_t->C: argument is not non-null

+

`cairo_surface_t' pointer

+

  argument: #<cpointer:cairo_t>

+ +

Unfortunately our old definition of ctx doesn’t have this tag:

+ +
+ + + + + +
> (cpointer-has-tag? ctx 'cairo_t)
+

#f

+ +

Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it’s also possible to update +existing pointers with a tag like this:

+ +
+ + + + + + + +
> (cpointer-push-tag! ctx 'cairo_t)
> (cpointer-has-tag? ctx 'cairo_t)
+

#t

+ +

Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).

+ +

Macros for reducing boilerplate

+ +

Now let’s start building the FFI bindings for the functions in the Cairo sample. +First, let’s go ahead and look at all of the types for the sample functions +from the C API docs:

+ +
+ + + + + + + + + + + +
void cairo_move_to (cairo_t *cr, double x, double y);
void cairo_line_to (cairo_t *cr, double x, double y);
void cairo_set_line_width (cairo_t *cr, double width);
void cairo_set_line_cap (cairo_t *cr, cairo_line_cap_t line_cap);
void cairo_stroke (cairo_t *cr);
+ +

Starting with cairo_move_to, we can set up a definition like we did +with cairo_create before:

+ +
+ + + +
+ + + + + + + + + + + +
> (define cairo-move-to
    (get-ffi-obj
     "cairo_move_to"
     cairo-lib
     (_fun _pointer _double _double -> _void)))
+ +

This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the ffi/unsafe/define library that help reduce the +verbosity. Here’s an alternative definition of cairo-move-to +using the define-ffi-definer form from +ffi/unsafe/define.

+ +
+ + + + + + + +
> (require ffi/unsafe/define)
> (define-ffi-definer define-cairo cairo-lib)
+ + + + + + + +
> (define-cairo cairo-move-to
    (_fun _cairo_t _double _double -> _void)
    #:c-id cairo_move_to)
+ +

As you can see, the define-ffi-definer form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +cairo_move_to) you also don’t need to supply the C name either.

+ +

The definitions for cairo_line_to, cairo_set_line_width, and +cairo_stroke aren’t very interesting, so I’ll just include them +below without comment:

+ +
+ + + + + + + +
+ + + + + + + +
> (define-cairo cairo-line-to
    (_fun _cairo_t _double _double -> _void)
    #:c-id cairo_line_to)
+ + + + + + + +
> (define-cairo cairo-set-line-width
    (_fun _cairo_t _double -> _void)
    #:c-id cairo_set_line_width)
+ + + + + + + +
> (define-cairo cairo-stroke
    (_fun _cairo_t -> _void)
    #:c-id cairo_stroke)
+ +

The cairo_set_line_cap case is more interesting because the type +cairo_line_cap_t is a C enumeration type. Racket’s FFI comes with +convenience forms for defining enumeration types— + though it’s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here’s a quote from the Barzilay and Orlovsky +paper on that:

+ +
+

Our design follows a simple principle: keep C-level +functionality to a minimum.

+ +

and specifically about enumerations:

+ +
+

For example, the C level part of our interface does not commit to a +specific implementation for enumerations — it simply exposes C integers.

+ +

To define an enumeration, we can use the _enum form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the cairo_line_cap_t type, it suffices to +just supply the cases as a list of symbols:

+ +
+ + + +
+ + + + + +
> (define _cairo_line_cap_t
    (_enum '(butt round square)))
+ +

The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It’s also possible to specify how the symbols map to integers more precisely +(see the docs on _enum for those details).

+ +

Given this type, we can specify the type for the line cap function:

+ +
+ + + +
+ + + + + + + +
> (define-cairo cairo-set-line-cap
    (_fun _cairo_t _cairo_line_cap_t -> _void)
    #:c-id cairo_set_line_cap)
+ +

Putting it all together

+ +

Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
(cairo-move-to ctx 50.0 75.0)
(cairo-line-to ctx 200.0 75.0)
 
(cairo-move-to ctx 50.0 125.0)
(cairo-line-to ctx 200.0 125.0)
 
(cairo-move-to ctx 50.0 175.0)
(cairo-line-to ctx 200.0 175.0)
 
(cairo-set-line-width ctx 30.0)
(cairo-set-line-cap ctx 'round)
(cairo-stroke ctx)
+ +

Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object bt. To see the +results of what we drew, we can just evaluate bt at the REPL. But it’s +a little nicer if we use the pict library to draw a frame around +it to distinguish it from the background:

+ +
+ + + + + + + +
> (require pict)
> (linewidth 2 (frame (bitmap bt)))
+

image

+ +

And we’re done! Of course, there is a lot more to the FFI. For example, I haven’t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I’m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/06/29/tutorial-racket-ffi-part-2/index.html b/blog/2016/06/29/tutorial-racket-ffi-part-2/index.html new file mode 100644 index 00000000..57a5716c --- /dev/null +++ b/blog/2016/06/29/tutorial-racket-ffi-part-2/index.html @@ -0,0 +1,880 @@ + + + + + + Tutorial: Racket FFI, Part 2 + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Tutorial: Racket FFI, Part 2

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 2 of my tutorial on using the Racket FFI. If you haven’t read +part 1 yet, you can find it +here. +Update: part 3 is also now available +here.

+ +

Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.

+ + +

First, here’s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#lang racket
(require racket/draw
         ffi/unsafe
         ffi/unsafe/define
         pict)
 
; bitmap magic
(define bt (make-bitmap 256 256))
(define bt-surface (send bt get-handle))
 
; C types
(define-cpointer-type _cairo_t)
(define-cpointer-type _cairo_surface_t)
(define _cairo_line_cap_t
  (_enum '(butt round square)))
 
(define cairo-lib (ffi-lib #f))
(define-ffi-definer define-cairo cairo-lib)
 
; the foreign functions
(define-cairo cairo-create
  (_fun _cairo_surface_t -> _cairo_t)
  #:c-id cairo_create)
(define-cairo cairo-move-to
  (_fun _cairo_t _double _double -> _void)
  #:c-id cairo_move_to)
(define-cairo cairo-line-to
  (_fun _cairo_t _double _double -> _void)
  #:c-id cairo_line_to)
(define-cairo cairo-set-line-width
  (_fun _cairo_t _double -> _void)
  #:c-id cairo_set_line_width)
(define-cairo cairo-stroke
  (_fun _cairo_t -> _void)
  #:c-id cairo_stroke)
(define-cairo cairo-set-line-cap
  (_fun _cairo_t _cairo_line_cap_t -> _void)
  #:c-id cairo_set_line_cap)
 
(define ctx (cairo-create bt-surface))
 
; Bitmap -> Pict
; a helper for displaying the bitmap
(define (show bt)
  (linewidth 2 (frame (bitmap bt))))
+ +

Dashes and array arguments

+ +

To start off, let’s look at another C example from the Cairo +samples page. +This time we will look at the "dash" example, which has a +use of an input array:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
double dashes[] = {50.0,  /* ink */
                   10.0,  /* skip */
                   10.0,  /* ink */
                   10.0   /* skip*/
                  };
int    ndash  = sizeof (dashes)/sizeof(dashes[0]);
double offset = -50.0;
 
cairo_set_dash (cr, dashes, ndash, offset);
cairo_set_line_width (cr, 10.0);
 
cairo_move_to (cr, 128.0, 25.6);
cairo_line_to (cr, 230.4, 230.4);
cairo_rel_line_to (cr, -102.4, 0.0);
cairo_curve_to (cr, 51.2, 230.4, 51.2, 128.0, 128.0, 128.0);
 
cairo_stroke (cr);
+ +

The most interesting function here is cairo_set_dash, which takes an +array argument. The only other new functions are cairo_rel_line_to +and cairo_curve_to which have very straightforward C types:

+ +
+ + + + + +
+ + + + + + + +
> (define-cairo cairo-rel-line-to
    (_fun _cairo_t _double _double -> _void)
    #:c-id cairo_rel_line_to)
+ + + + + + + + + + + + + + + +
> (define-cairo cairo-curve-to
    (_fun _cairo_t
          _double _double
          _double _double
          _double _double
          -> _void)
    #:c-id cairo_curve_to)
+ +

Meanwhile, the C type signature for cairo_set_dash from the Cairo +docs looks like this:

+ +
+ + + + + + + + + +
void cairo_set_dash (cairo_t *cr,
                     const double *dashes,
                     int num_dashes,
                     double offset);
+ +

Something to note about the arguments is that num_dashes +encodes the length of the array dashes. This will come up later when +we want to make the C type for this function more convenient.

+ +

On the Racket side, it’s natural to represent the array of dashes as either a +list or vector of numbers. Given that, a fairly literal translation of +the C type above might look like the following:

+ +
+ + + +
+ + + + + + + + + + + + + + + +
> (define-cairo cairo-set-dash
    (_fun _cairo_t
          (_list i _double)
          _int
          _double
          -> _void)
    #:c-id cairo_set_dash)
+ +

This type includes a type constructor we haven’t seen yet: _list. +This is a so-called custom function type that has special meaning +inside of a _fun type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.

+ +

Since we only want to provide a list from the Racket side to C, we’ll use +the i input mode. We can then call the function like this:

+ +
+ + + + + + + + + +
(cairo-set-dash ctx
                (list 50.0 10.0 10.0 10.0)
                4
                -50.0)
+ +

Note that because of how we defined the type of cairo-set-dash we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it’s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.

+ +

Luckily, the _fun type constructor actually lets you do this with the +(name : type) syntax for naming arguments in combination with the +(type = expr) syntax for supplying computed arguments:

+ +
+ + + +
+ + + + + + + + + + + + + + + + + + + +
> (define-cairo cairo-set-dash
    (_fun _cairo_t
          ; name this argument for later uses in the type
          [dashes : (_list i _double)]
          ; a computed argument position
          [_int = (length dashes)]
          _double
          -> _void)
    #:c-id cairo_set_dash)
+ +

When a computed argument is specified with a =, it’s not necessary to provide +the argument on the Racket side. So cairo-set-dash is now an arity 3 +function that can be called like this:

+ +
+ + + + + + + +
(cairo-set-dash ctx
                (list 50.0 10.0 10.0 10.0)
                -50.0)
+ +

This means we’ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it’s also possible to use Racket vectors instead of lists by using +the _vector type constructor.

+ +

Putting it all together, we can reproduce the dashes example like this:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
(define dashes '(50.0 10.0 10.0 10.0))
(define offset -50.0)
+ + + +
(cairo-set-dash ctx dashes offset)
(cairo-set-line-width ctx 10.0)
+ + + +
(cairo-move-to ctx 128.0 25.6)
(cairo-line-to ctx 230.4 230.4)
(cairo-rel-line-to ctx -102.4 0.0)
+ + + + + +
(cairo-curve-to ctx 51.2 230.4 51.2
                    128.0 128.0 128.0)
+ + + +
(cairo-stroke ctx)
+ + + +
(show bt)
+

image

+ +

Result arguments and C structs

+ +

For some more advanced FFI hacking, let’s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:

+ +
+ + + + + + + +
> (define txt-bt (make-bitmap 256 256))
> (define txt-surface (send txt-bt get-handle))
> (define txt-ctx (cairo-create txt-surface))
+ +

Our challenge is to make a Racket function that takes a string (let’s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let’s just scale the +text to fit the width and assume the height will be okay.

+ +

To implement the key step of measuring the text size, we can use the +cairo_text_extents +function. Its type signature is as follows:

+ +
+ + + + + + + + + +
void
cairo_text_extents (cairo_t *cr,
                    const char *utf8,
                    cairo_text_extents_t *extents);
+ +

The interesting part of this signature is that +cairo_text_extents_t +is a struct type:

+ +
+ + + + + + + + + + + + + + + + + + + +
/* from the Cairo docs */
typedef struct {
    double x_bearing;
    double y_bearing;
    double width;
    double height;
    double x_advance;
    double y_advance;
} cairo_text_extents_t;
+ +

We haven’t yet seen how to handle C structs with the FFI, but it’s not +too tricky. Support for C structs comes built-in and will look familiar +if you’re used to Racket structs. We can directly translate the documented +definition above into a define-cstruct declaration:

+ +
+ + + + + +
; the leading underscore is mandatory
+ + + + + + + + + + + + + + + +
> (define-cstruct _cairo_text_extents_t
    ([x-bearing _double]
     [y-bearing _double]
     [width _double]
     [height _double]
     [x-advance _double]
     [y-advance _double]))
+ +

This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:

+ +
+ + + + + + + + + + + + + + + + + +
> _cairo_text_extents_t
+

#<ctype>

; pointer to struct
> _cairo_text_extents_t-pointer
+

#<ctype>

; allows NULL pointer
> _cairo_text_extents_t-pointer/null
+

#<ctype>

+ +

Along with functions that look like regular Racket struct operations:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
; a struct constructor
> make-cairo_text_extents_t
+

#<procedure:make-cairo_text_extents_t>

; a field selector
> cairo_text_extents_t-width
+

#<procedure:cairo_text_extents_t-width>

; a predicate for the struct
> cairo_text_extents_t?
+

#<procedure:^TYPE?>

; a field mutation function
> set-cairo_text_extents_t-width!
+

#<procedure:set-cairo_text_extents_t-width!>

+ +

With the struct type defined, it’s easy to come up with a rudimentary +interface for cairo-text-extents:

+ +
+ + + +
+ + + + + + + + + + + + + +
> (define-cairo cairo-text-extents
    (_fun _cairo_t
          _string
          _cairo_text_extents_t-pointer
          -> _void)
    #:c-id cairo_text_extents)
+ +

In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:

+ +
+ + + + + + + + + +
+ + + + + + + +
> (define extents
    (make-cairo_text_extents_t
     0.0 0.0 0.0 0.0 0.0 0.0))
+ + + + + +
> (cairo-text-extents
   txt-ctx "hello world" extents)
> (cairo_text_extents_t-width extents)
+

54.0

+ +

This style of programming feels awfully imperative though. Since we’re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the cairo-text-extents FFI wrapper +by combining named arguments, a new _ptr type constructor, +and a neat feature of _fun that lets you customize +the return result:

+ +
+ + + +
+ + + + + + + + + + + + + + + + + + + + + +
> (define-cairo cairo-text-extents*
    (_fun _cairo_t
          _string
          ; named args and _ptr
          [ext : (_ptr o _cairo_text_extents_t)]
          ; the return result of the C function
          -> _void
          ; custom return result for the wrapper
          -> ext)
    #:c-id cairo_text_extents)
+ +

The _ptr constructor works like the _list constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the o mode to _ptr. +In output mode, this type will automatically allocate a new instance of the type +(using the malloc function) and arrange for it to be passed in as +a pointer.

+ +

The strangest part of this example is that there are now two uses of the +-> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.

+ +

Using this new version of the wrapper is much simpler:

+ +
+ + + + + +
+ + + + + +
> (cairo_text_extents_t-width
   (cairo-text-extents* txt-ctx "hello world"))
+

54.0

+ +

With that in hand, it’s pretty easy to write the function we set out to write:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
(define-cairo cairo-show-text
  (_fun _cairo_t _string -> _void)
  #:c-id cairo_show_text)
 
(define-cairo cairo-scale
  (_fun _cairo_t _double _double -> _void)
  #:c-id cairo_scale)
 
; String -> Void
; draws a string scaled horizontally
(define (fit-text str)
  (define padding 20)
  (cairo-move-to txt-ctx (/ padding 2.0) 128.0)
  (define extents
    (cairo-text-extents* txt-ctx str))
  (define x-bearing
    (cairo_text_extents_t-x-bearing
     extents))
  (define width
    (cairo_text_extents_t-width
     extents))
  (define scale (/ (- 256.0 padding)
                   (+ x-bearing width)))
  (cairo-scale txt-ctx scale scale)
  (cairo-show-text txt-ctx str))
+ +

And to conclude part 2 of this tutorial, here’s an example use +of the new fit-text function:

+ +
+ + + + + + + +
> (fit-text "Saluton, Mondo / Hallo, mundo")
> (show txt-bt)
+

image

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/07/11/tutorial-racket-ffi-part-3/index.html b/blog/2016/07/11/tutorial-racket-ffi-part-3/index.html new file mode 100644 index 00000000..7238ed9a --- /dev/null +++ b/blog/2016/07/11/tutorial-racket-ffi-part-3/index.html @@ -0,0 +1,1127 @@ + + + + + + Tutorial: Racket FFI, part 3 + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Tutorial: Racket FFI, part 3

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +here +and part 2 +here.

+ +

In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.

+ + +

As in the second post, let’s start with some prologue code that establishes +the definitions from the previous two posts. But first, I’m getting tired of +writing the #:c-id identifier notation for the underscored C function +names.

+ +

Instead, let’s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:

+ +
+ + + +
$ raco pkg install ffi-definer-convention
+ +

or you can just execute the following snippet in Racket:

+ +
+ + + + + + + +
> (require pkg)
> (pkg-install-command #:skip-installed #t "ffi-definer-convention")
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+

Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/

+

Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org

+

Downloading repository git://github.com/takikawa/racket-ffi-definer-convention

+

raco setup: version: 7.9

+

raco setup: platform: x86_64-linux [3m]

+

raco setup: target machine: racket

+

raco setup: installation name: 7.9

+

raco setup: variants: 3m

+

raco setup: main collects: /usr/share/racket/collects

+

raco setup: collects paths:

+

raco setup:   /usr/share/racket/collects

+

raco setup: main pkgs: /usr/share/racket/pkgs

+

raco setup: pkgs paths:

+

raco setup:   /usr/share/racket/pkgs

+

raco setup:   /root/.local/share/racket/7.9/pkgs

+

raco setup: links files:

+

raco setup:   /usr/share/racket/links.rktd

+

raco setup:   /root/.local/share/racket/7.9/links.rktd

+

raco setup: main docs: /usr/share/racket/doc

+

raco setup: --- updating info-domain tables ---                    [22:20:56]

+

raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd

+

raco setup: --- pre-installing collections ---                     [22:20:56]

+

raco setup: --- installing foreign libraries ---                   [22:20:56]

+

raco setup: --- installing shared files ---                        [22:20:56]

+

raco setup: --- compiling collections ---                          [22:20:56]

+

raco setup: --- parallel build using 4 jobs ---                    [22:20:56]

+

raco setup: 3 making: <pkgs>/ffi-definer-convention

+

raco setup: --- creating launchers ---                             [22:20:57]

+

raco setup: --- installing man pages ---                           [22:20:57]

+

raco setup: --- building documentation ---                         [22:20:57]

+

raco setup: 3 running: <pkgs>/ffi-definer-convention/ffi-definer-convention.scrbl

+

raco setup: 3 rendering: <pkgs>/ffi-definer-convention/ffi-definer-convention.scrbl

+

raco setup: 2 rendering: <pkgs>/racket-index/scribblings/main/user/local-redirect.scrbl

+

raco setup: 1 rendering: <pkgs>/racket-index/scribblings/main/user/release.scrbl

+

raco setup: 0 rendering: <pkgs>/racket-index/scribblings/main/user/search.scrbl

+

raco setup: 0 rendering: <pkgs>/racket-index/scribblings/main/user/start.scrbl

+

raco setup: --- installing collections ---                         [22:21:03]

+

raco setup: --- post-installing collections ---                    [22:21:03]

+ +

This will install the package and compile its contents. If you’re curious, the +docs for the package are available +here.

+ +

Note: if you’ve never installed a package before, you may want to glance +at the +package system docs. +A tl;dr of packages is that they bundle Racket collections, which are +sets of modules that you can refer to in a location-independent fashion such as +pict or racket/list.

+ +

Anyhow, here’s the prologue code:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#lang racket
(require racket/draw
         ffi/unsafe
         ; avoid conflict with below
         (except-in ffi/unsafe/define
                    define-ffi-definer)
         ; the new 3rd-party pkg
         ffi-definer-convention
         pict)
 
; C types
(define-cpointer-type _cairo_t)
(define-cpointer-type _cairo_surface_t)
(define _cairo_line_cap_t
  (_enum '(butt round square)))
 
(define cairo-lib (ffi-lib #f))
(define-ffi-definer define-cairo cairo-lib
  ; describes how to transform from
  ; Racket to C ids
  #:make-c-id convention:hyphen->underscore)
 
; the foreign functions
; note lack of #:c-id keyword arguments
(define-cairo cairo-create
  (_fun _cairo_surface_t -> _cairo_t))
(define-cairo cairo-move-to
  (_fun _cairo_t _double _double -> _void))
(define-cairo cairo-line-to
  (_fun _cairo_t _double _double -> _void))
(define-cairo cairo-set-line-width
  (_fun _cairo_t _double -> _void))
(define-cairo cairo-stroke
  (_fun _cairo_t -> _void))
(define-cairo cairo-set-line-cap
  (_fun _cairo_t _cairo_line_cap_t -> _void))
 
; (_cairo_t -> Void) -> Pict
; do some drawing and give us the pict
(define (do-cairo f)
  (define bt (make-bitmap 256 256))
  (define bt-surface (send bt get-handle))
  (f (cairo-create bt-surface))
  (linewidth 2 (frame (bitmap bt))))
+ +

Notice that the define-cairo forms don’t have any #:c-id keywords +anymore. Instead, the prologue code uses an overriden define-ffi-definer +from my package that supports a #:make-c-id keyword that lets you specify +a naming convention to follow.

+ +

Also, instead of creating a single bitmap and drawing into it, we now have a +do-cairo function that takes a drawing function. When called, +do-cairo will call the given function with a new bitmap object and +return the result.

+ +

Now let’s get to the main point of this blog post. Let’s say that we want to play +with Cairo path +objects this time. A path is +defined +as a struct with the following structure:

+ +
+ + + + + + + + + + + +
typedef struct {
    cairo_status_t status;
    cairo_path_data_t *data;
    int num_data;
} cairo_path_t;
+ +

To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it’s +useful to define C types for the types of values in the path struct’s fields. First, +let’s specify that a cairo_status_t is an integer type:

+ +
+ + + +
> (define _cairo_status_t _int)
+ +

It’s actually an enum, but for the examples in this post we don’t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +path data objects. +Each path data object is a cairo_path_data_t, +which is specified with a C union:

+ +
+ + + + + + + + + + + + + + + + + + + +
union _cairo_path_data_t {
    struct {
        cairo_path_data_type_t type;
        int length;
    } header;
    struct {
        double x, y;
    } point;
};
+ +

Helpfully, the FFI library comes with support for unions with the +_union type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It’s pretty +straightforward to specify this type too:

+ +
+ + + + + + + +
; the path data type is just an enum
+ + + + + +
> (define _cairo_path_data_type_t
    (_enum '(move-to line-to curve-to close-path)))
+ + + + + + + + + + + + + +
> (define _cairo_path_data_t
    (_union ; the header case
            (_list-struct _cairo_path_data_type_t
                          _int)
            ; the point case
            (_list-struct _double _double)))
+ +

There’s a new type constructor here so let me explain that first. +The _list-struct constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +define-cstruct, this constructor doesn’t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.

+ +

Each of the path data structs in the path data array will be manipulated with +the _cairo_path_data_t type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:

+ +
+ + + + + + + + + +
; create a union from a list of doubles
+ + + + + + + + + + + + + +
> (define a-union-val
    (cast (list 1.3 5.8)
          ; source type
          (_list-struct _double _double)
          ; target type
          _cairo_path_data_t))
> a-union-val
+

#<union>

+ +

This snippet first construct a union object (via the _cairo_path_data_t +type) using a cast. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it’s an easy way +to generate a union object.

+ +

The second line shows that a union prints as an opaque object. You can’t do +anything with a union in Racket unless you project it to one of the sub-cases with +the union-ref function. +This projection is unsafe, in the sense that if you don’t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.

+ +

More concretely, let’s see what happens if we try to extract a value out of the +union both correctly and incorrectly:

+ +
+ + + + + + + + + + + + + + + + + +
; correct (matches construction)
; cases are zero-indexed and ordered as written
> (union-ref a-union-val 1)
+

'(1.3 5.8)

; incorrect, error
> (union-ref a-union-val 0)
+

enum:int->_cairo_path_data_type_t: expected a known

+

#<ctype:ufixint>, got: 3435973837

+ +

Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.

+ +

With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.

+ +

With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it’s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.

+ +

So before moving on, let’s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we’ll figure out how to +deal with the array itself.

+ +

Some low-level operations

+ +

Since we still don’t have a C type for cairo_path_t, let’s go ahead +and make a simple one where we punt on the work of specifying the array +type:

+ +
+ + + +
+ + + + + + + + + +
> (define _simple_cairo_path_t
    (_list-struct _cairo_status_t
                  _pointer
                  _int))
+ +

In this type, we have specified the array as a bare _pointer. +For some added safety, we could also use something like +(_cpointer 'cairo_status_t), which sets up a tagged pointer +type like we saw in the first blog post with +define-cpointer-type.

+ +

We’ve seen the _pointer type before, but haven’t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.

+ +

Before we get to that, let’s go ahead and set up an FFI binding for +cairo_copy_path so that we can obtain a path struct to manipulate:

+ +
+ + + + + + + + + + + + + +
+ + + + + +
> (define-cairo cairo-copy-path
    (_fun _cairo_t -> _pointer))
> (define a-path #f)
+ + + + + + + + + + + + + + + + + + + + + + + + + +
> (do-cairo (λ (ctx)
              ; Do stuff to make the current
              ; path non-empty
              (cairo-move-to ctx 50.0 50.0)
              (cairo-line-to ctx 206.0 206.0)
              (cairo-move-to ctx 50.0 206.0)
              (cairo-line-to ctx 115.0 115.0)
              ; Get the current path
              (set! a-path (cairo-copy-path ctx))
              ; Stroke clears the path
              ; so do it last
              (cairo-stroke ctx)))
+

image

> a-path
+

#<cpointer>

+ +

Note that cairo-copy-path gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is ptr-ref, which +lets you dereference a pointer and access it at some concrete C type.

+ +

Note: the ptr-ref function also takes an optional +offset argument which we will be used in an example later.

+ +

For example, we can use a-path as a _simple_cairo_path_t:

+ +
+ + + + + + + +
+ + + + + +
> (define simple-path
    (ptr-ref a-path _simple_cairo_path_t))
> simple-path
+

'(0 #<cpointer> 8)

+ +

And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use ptr-ref again with an array type:

+ +
+ + + + + + + +
+ + + + + + + + + + + + + +
> (define array
    (ptr-ref ; the pointer
             (second simple-path)
             (_array/list _cairo_path_data_t
                          ; length field
                          (third simple-path))))
> array
+

'(#<union> #<union> #<union> #<union> #<union> #<union> #<union> #<union>)

+ +

The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:

+ +
+ + + + + + + + + + + + + + + +
> (union-ref (first array) 0)
+

'(move-to 2)

> (union-ref (second array) 1)
+

'(50.0 50.0)

; nonsense data here, wrong union case
> (union-ref (third array) 1)
+

'(4.2439915824246e-314 0.0)

+ +

One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.

+ +

An alternative is to define a custom C type that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don’t need to think about applying helper functions and +dereferencing pointers.

+ +

Custom C types

+ +

I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.

+ +

In this way, it’s possible to conduct interesting conversions, such as +dereferencing union objects automatically.

+ +

Now let’s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.

+ +

First, we’ll start by defining a struct type for the Racket representation +of Cairo paths:

+ +
+ + + +
+ + + + + + + +
> (struct cairo-path (ptr)
    #:property prop:sequence
    (λ (p) (in-cairo-path p)))
+ +

The representation will store one field ptr which, as the name +suggests, will store a pointer value. We’ll see what to do with this +pointer later.

+ +

This definition uses a structure type property to make instances of +cairo-path automatically work as sequences. This means that you +can iterate over them with a for loop or apply sequence-ref +on them. The property takes a function that takes an instance of the struct +type itself (here p) and that returns a sequence.

+ +

We’ll later define the in-cairo-path function that will actually +construct the relevant sequence for us. For now, let’s see how to construct +the C type given this struct type:

+ +
+ + + +
+ + + + + + + + + + + + + + + + + + + + + + + +
> (define _cairo_path_t
    (let ()
      ; Extract pointer out of representation
      (define (racket->c rkt)
        (cairo-path-ptr rkt))
      ; Just apply the Racket constructor
      (define (c->racket cobj)
        (cairo-path cobj))
      (make-ctype _pointer
                  racket->c
                  c->racket)))
+ +

The base type for this _cairo_path_t is a _pointer type. Since +the Cairo API returns pointers to new path values, it’s hard to avoid using some +kind of pointer type as the base type here.

+ +

This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we’ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.

+ +

The real work is done by the helper function that makes a +cairo-path instance work as a sequence.

+ +

Starting top-down, let’s look at the definition of in-cairo-path:

+ +
+ + + + + +
; Cairo-Path -> Sequence
+ + + + + + + + + + + + + + + + + + + + + + + + + +
> (define (in-cairo-path path)
    (define pp (cairo-path-ptr path))
    (match-define
      (list _ array-ptr len)
      (ptr-ref pp _simple_cairo_path_t))
    (make-do-sequence
      (λ ()
        (values (pos->element array-ptr)
                (next-pos array-ptr)
                0
                (λ (pos) (< pos len))
                #f #f))))
+ +

The first thing the function does is extract the pointer out of the +representation, and then immediately calls ptr-ref on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.

+ +

Note: in case you’re not very familiar with Racket pattern matching, the +match-define form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml’s let statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +docs.

+ +

After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +make-do-sequence function. Essentially, make-do-sequence +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.

+ +

Note: technically make-do-sequence actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it’s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.

+ +

In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the pos->element function +and its helpers:

+ +
+ + + + + + + + + +
; CPointer -> Integer -> Element
+ + + + + + + + + + + + + + + + + + + + + + + + + +
> (define ((pos->element ptr) pos)
    ; Extract the data path header
    (define header
      (union-ref
       (ptr-ref ptr _cairo_path_data_t pos)
       0))
    (define type   (first header))
    ; Length includes header, so subtract 1
    (define len    (sub1 (second header)))
    (define pos*   (add1 pos))
    (define points (get-points ptr pos* len))
    (cons type points))
; CPointer Integer Integer -> (Listof Data)
+ + + + + + + + + + + + + + + +
> (define (get-points ptr pos num-points)
    (for/list ([i (in-range num-points)])
      (union-ref (ptr-ref ptr
                          _cairo_path_data_t
                          ; offset argument
                          (+ pos i))
                 1)))
+ +

This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in get-points +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.

+ +

Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:

+ +
+ + + +
+ + + + + + + + + + + + + + + +
> (define ((next-pos ptr) pos)
    (define header
      (union-ref
       (ptr-ref ptr _cairo_path_data_t pos)
       0))
    (define len (second header))
    (+ len pos))
+ +

Note that determining the end of the sequence is very easy. It’s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression (λ (pos) (< pos len)).

+ +

Now we can try using a path as a sequence:

+ +
+ + + + + + + + + +
+ + + + + +
> (define-cairo cairo-copy-path
    (_fun _cairo_t -> _cairo_path_t))
+ + + + + + + + + + + + + + + + + + + + + +
> (do-cairo (λ (ctx)
              (cairo-move-to ctx 50.0 50.0)
              (cairo-line-to ctx 206.0 206.0)
              (cairo-move-to ctx 50.0 206.0)
              (cairo-line-to ctx 115.0 115.0)
              (define path (cairo-copy-path ctx))
              ; Using path as a sequence
              (for ([elem path])
                (displayln elem))
              (cairo-stroke ctx)))
+ + + + + + + + + +
+

(move-to (50.0 50.0))

+

(line-to (206.0 206.0))

+

(move-to (50.0 206.0))

+

(line-to (115.0 115.0))

+

image

+ +

Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the _array/list type.

+ +

That concludes part 3 of the FFI tutorial. Hopefully you’re now equipped +to deal with union types and custom C types. If not, see the +FFI reference +for more details on +unions +and +custom C types.

+ +

Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/08/02/tutorial-zero-to-sixty-in-racket/index.html b/blog/2016/08/02/tutorial-zero-to-sixty-in-racket/index.html new file mode 100644 index 00000000..d05db2a5 --- /dev/null +++ b/blog/2016/08/02/tutorial-zero-to-sixty-in-racket/index.html @@ -0,0 +1,1247 @@ + + + + + + Tutorial: Zero to Sixty in Racket + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Tutorial: Zero to Sixty in Racket

+

+ :: Racket, tutorial

+

By: Ben Greenman

+
+ +

Racket is excellent for incrementally growing scripts into full-fledged programs. +This post steps through the evolution of one small program and highlights the + Racket tools that enable incremental advances.

+ + +

+ +
Why should anyone use Racket? +There are two reasons: +
+ +
+
    +
  1. +

    You have a problem that can only be solved with Racket’s language-building tools

    2. +
  2. +

    Racket is a nice language to program in. +(Has lexical scope, parentheses, active users...)

+ +

My favorite part of Racket is how it supports a certain development style of + evolving scripts into programs. +When I start coding (after design, before debugging), I can focus the problem at hand. +Next come examples, unit tests, and types to be sure the solution is correct. +Finally, I worry about aesthetics, efficiency, and how the solution can be used as a library in a larger context.

+ +

Bottom line: with Racket, my coding is aligned with my priorities. +And as I transition from "no code" to "working code" to "robust code" to "re-usable code", + the program is almost always runnable.

+ +

Problem: A KWIC Index Production System

+ +

A KWIC index system +reads input from a file, +divides each line of the file into whitespace-separated words, +and outputs (in alphabetical order) all circular shifts of all lines.

+ +

The first circular shift of a line "A B C" is the line "B C A". +The second circular shift is "C A B".

+ +

Building a KWIC index is a historical problem. +According to D.L. Parnas (1972):

+ +
+

Except under extreme circumstances (huge data base, no supporting software) +such a system could be implemented by a good programmer within a week or two.

+ +

See also: Yannis’s Law.

+ +

Today, I bet only Agda and Scratch + programmers would need the full two weeks. +We’ll be done in 20 minutes.

+ +

A Script

+ +

To start, open a file and type:

+ +
+ + + +
#lang racket
+ +

You can name the file anything, like kwic.rkt or rkt.kwic or foo. +Racket doesn’t care, + but it does need the #lang line to read the contents of the file.

+ +

Though, you should use the .rkt extension.

+ +

The first part of the solution is a function to read input from a file into a + list of strings for further processing. +The built-in function file->lines + does exactly this, but we’ll for-loop instead.

+ +
+ + + + + + + + + + + +
(define (kwic-read filename)
  (with-input-from-file filename
    (λ ()
      (for/list ([line (in-lines)])
        line))))
+ +

When called with a filename like "heart-of-darkness.txt", the function + uses for/list to build a list of lines by reading from a port + with in-lines. +The port is the data from filename, thanks to with-input-from-file.

+ +

Next is a function to convert a list of strings into a list of lists of words. +Here we’ll just use library functions.

+ +
+ + + + + +
(define (kwic-split lines)
  (map string-split lines))
+ +

By default, string-split divides a string into a list of whitespace-separated substrings. +You can always supply a different delimiter, or use regexp-split + to divide by a regular expression.

+ +

Two tasks left! +First we generate all circular shifts for a list of strings words + by folding up a list with one shift of words for each word.

+ +
+ + + + + + + + + + + + + + + + + +
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(define (all-circular-shifts words)
  (for/fold ([all-shifts (list words)])
            ([i (in-range 1 (length words))])
    (cons (circular-shift (first all-shifts))
          all-shifts)))
+ +

Second, we alphabetize and print the shifts.

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
(define (alphabetize all-shifts)
  (sort all-shifts shift<?))
 
(define (shift<? shift1 shift2)
  (match* (shift1 shift2) ; destruct multiple values
   [('() _) ; first list empty, don't care about second
    #t]
   [(_ '()) ; first list non-empty, second empty
    #f]
   [((cons s1 shift1-rest) (cons s2 shift2-rest))
    (or (string<? s1 s2)
        (and (string=? s1 s2)
             (shift<? shift1-rest shift2-rest)))]))
 
(define (kwic-display all-sorted-shifts)
  (define (display-words words)
    (display (first words))
    (for ([word (in-list (cdr words))])
      (display " ")
      (display word))
    (newline))
  ; for-each is like map, but returns (void)
  (for-each display-words all-sorted-shifts))
+ +

Gluing it all together, here’s the full script (with type annotations in comments).

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#lang racket
 
; type Words = (Listof String)
; type Lines = (Listof Words)
 
; Path-String -> (Listof String)
(define (kwic-read filename)
  (with-input-from-file filename
    (λ ()
      (for/list ([line (in-lines)])
        line))))
 
; (Listof String) -> Lines
(define (kwic-split lines)
  (map string-split lines))
 
; Words -> (Listof Words)
(define (all-circular-shifts words)
  (for/fold ([all-shifts (list words)])
            ([i (in-range 1 (length words))])
    (cons (circular-shift (first all-shifts))
          all-shifts)))
 
; Move first element to last position
; Words -> Words
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
; Lines -> Lines
(define (alphabetize all-shifts)
  (sort all-shifts shift<?))
 
; Lexicographic order on equal-length lists of words
; Words Words -> Boolean
(define (shift<? shift1 shift2)
  (match* (shift1 shift2)
   [('() _) ; first list empty, don't care about second
    #t]
   [(_ '()) ; first list non-empty, second empty
    #f]
   [((cons s1 shift1-rest) (cons s2 shift2-rest))
    (or (string<? s1 s2)
        (and (string=? s1 s2)
             (not (null? shift1-rest))
             (shift<? shift1-rest shift2-rest)))]))
 
; Lines -> Void
(define (kwic-display all-sorted-shifts)
  (define (display-words words)
    (display (first words))
    (for ([word (in-list (cdr words))])
      (display " ")
      (display word))
    (newline))
  (for-each display-words all-sorted-shifts))
 
; Lines -> (Listof Lines)
(define (all-circular-shifts* lines)
  (map all-circular-shifts lines))
 
; Path-String -> Void
(define (kwic-index file-name)
  (define all-lines (kwic-split (kwic-read file-name)))
  (define all-shifts (append* (all-circular-shifts* all-lines)))
  (kwic-display (alphabetize all-shifts)))
 
; End-to-end test
; -> Void
(define (run-test)
  (define test-file "test.txt")
  ; Make a file and test
  (unless (file-exists? test-file)
    (with-output-to-file test-file
      (λ ()
        (displayln "imagine if this")
        (displayln "took 2 weeks to write"))))
  (kwic-index test-file))
 
(run-test)
+ +

Running the file should print:

+ +
+ + + + + + + + + + + + + + + + + +
2 weeks to write took
if this imagine
imagine if this
this imagine if
to write took 2 weeks
took 2 weeks to write
weeks to write took 2
write took 2 weeks to
+ +

Testing and Submodules

+ +

Any top-level expressions in a file can work as unit tests. +The equal? statement below checks whether the first circular shift + of '("A" "B" "C") is '("B" "C" "A").

+ +
+ + + + + + + + + +
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(equal? (circular-shift '("A" "B" "C")) '("B" "C" "A"))
+ +

Running the file now prints #t to the console, meaning the test passed. +We can use error or raise-user-error to make failures easier + to notice. +Or we can use the RackUnit testing library.

+ +
+ + + + + + + + + + + + + + + +
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(require rackunit) ; import the testing library
(check-equal?
  (circular-shift '("A" "B" "C"))
  '("B" "C" "A"))
+ +

These tests run each time the module does. +If you prefer to run tests only in a specific context, and not when the + module is run or imported as a library, you can move them to a separate + file or into a submodule.

+ +
+ + + + + + + + + + + + + + + + + +
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(module+ test ; Open a submodule named 'test'
  (require rackunit)
  (check-equal?
    (circular-shift '("A" "B" "C"))
    '("B" "C" "A")))
+ +

Running the module normally via racket kwic.rkt will not run code + in the submodule. +Instead, use raco test to run the tests.

+ +
+ + + + + + + +
>  raco test kwic.rkt
raco test: (submod "kwic.rkt" test)
1 test passed
+ +

The reason we used module+, instead of Racket’s module and module* + forms is that module+ inherits the language and namespace of its + containing module and can be incrementally extended. +This way, we can keep tests near the relevant code.

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
(module+ test
  (require rackunit))
 
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(module+ test
  (check-equal?
    (circular-shift '("A" "B" "C"))
    '("B" "C" "A")))
 
(define (alphabetize all-shifts)
  (sort all-shifts shift<?))
 
(module+ test
  (check-equal?
    (alphabetize '(("racket" "is")
                   ("as")
                   ("racket" "does")))
    '(("as")
      ("racket" "does")
      ("racket" "is"))))
+ +

RackUnit in a + separate file or test submodule is the unofficial standard for testing + Racket programs.

+ +

Recognizing Patterns, Avoiding Repetition

+ +

Every unit test we’ve written uses check-equal?.

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
(module+ test
  (check-equal?
    (kwic-split '())
    '())
  (check-equal?
    (kwic-split '("hello    world"))
    '(("hello" "world")))
  (check-equal?
    (kwic-split '(" lost " " in " "space"))
    '(("lost") ("in") ("space")))
  (check-equal?
    (kwic-split '("something"))
    '()))
+ +

These tests follow a simple pattern that we can express as a syntax rule.

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
(module+ test
  (define-syntax-rule (check-equal?* [i o] ...)
    (begin
      (check-equal? i o)
      ...))
 
  (check-equal?*
    [(kwic-split '())
     '()]
    [(kwic-split '("hello    world"))
     '(("hello" "world"))]
    [(kwic-split '(" out " " in " "the ozone"))
     '(("out") ("in") ("the" "ozone"))]
    [(kwic-split '("something"))
     '()]))
+ +

The ... are not pseudocode! +They denote Kleene-star repetition, like a sextile ("*") in a regular expression. +In this case, the input pattern is a sequence of lists with two S-expressions, i and + o. +Uses of i and o in the rule must be followed by one ... to splice + the captured S-expressions into the result.

+ +

Many languages offer higher-order functions and polymorphism to abstract common + behaviors. +Syntax extensions are a different way to avoid repeating yourself. +After 30 years, we are still discovering what syntax extensions are useful for.

+ +

See this recent Racket mailing list post for some applications.

+ +

Adding Static Types

+ +

Changing the #lang line to typed/racket adds static type-checking to our program. +If we only change the language and run the code as-is, there will be type errors. +But we can use submodules again to incrementally check our design with types.

+ +

Note: typed regions + are another way to embed typed code into untyped contexts.

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#lang racket
 
 (module t typed/racket
   ; Need to annotate:
   ; - function parameters
   ; - for-loop return types
 
   (: kwic-read : Path-String -> (Listof String))
   (define (kwic-read filename)
     (with-input-from-file filename
       (λ ()
         (for/list ([line (in-lines)])
                   : (Listof String)
           line))))
 
   ; Next migration step: move other untyped functions here
 
   (provide (all-defined-out)))
 (require 't)
 
 (define (kwic-split lines)
   (map string-split lines))
 
 ; <rest of file omitted>
+ +

After scooping all functions into the Typed Racket bubble, we can remove the + submodule declaration and change #lang racket to #lang typed/racket.

+ +

Finally, a Library

+ +

Other modules can import our functions if we use a provide statement. +By convention, exports belong at the top of a file.

+ +
+ + + + + + + + + + + + + +
 
#lang typed/racket
 
(provide kwic-index)
 
; <definitions here>
+ +

Then any typed or untyped module can use kwic-index by writing + (require "kwic.rkt").

+ +

As a finishing touch, we can use the racket/cmdline library + inside a main submodule to give a basic front-end interface. +Similar to module+ test, a module+ main declares code that + inherits the file’s bindings and language but is only run when the program + is executaed.

+ +

Here is the complete typed and tested code listing. +The main submodule is at the bottom.

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#lang typed/racket
(module+ test
  (require typed/rackunit)
 
  (define-syntax-rule (check-equal?* [i o] ...)
    (begin
      (check-equal? i o)
      ...)))
 
(define-type Words (Listof String))
(define-type Lines (Listof Words))
 
(: kwic-read : Path-String -> (Listof String))
(define (kwic-read filename)
  (with-input-from-file filename
    (λ ()
      (for/list ([line (in-lines)])
                : (Listof String)
        line))))
 
(module+ test
  (let ([tmpfile (make-temporary-file)])
    (with-output-to-file tmpfile #:exists 'replace
      (λ ()
        (displayln "The Nellie,")
        (displayln "a cruising yawl,")
        (displayln "swung to her anchor without a flutter of sails,")
        (displayln "and was at rest.")))
    (define actual (kwic-read tmpfile))
    (define expect (file->lines tmpfile))
    (delete-file tmpfile)
    (check-equal? actual expect)))
 
(: kwic-split : (Listof String) -> Lines)
(define (kwic-split lines)
  (map #{string-split :: (String -> Words)} lines))
 
(module+ test
  (check-equal?*
    [(kwic-split '())
     '()]
    [(kwic-split '("hello    world"))
     '(("hello" "world"))]))
 
; Move first element to last position
(: circular-shift : Words -> Words)
(define (circular-shift words)
  (append (rest words) (list (first words))))
 
(module+ test
  (check-equal?*
    [(circular-shift '("A" "B" "C"))
     '("B" "C" "A")]))
 
(: all-circular-shifts : Words -> (Listof Words))
(define (all-circular-shifts words)
  (for/fold ([all-shifts (list words)])
            ([i (in-range 1 (length words))])
            : (Listof Words)
    (cons (circular-shift (first all-shifts))
          all-shifts)))
 
(module+ test
  (check-equal?*
    [(all-circular-shifts '("A" "B" "C"))
     '(("C" "A" "B") ("B" "C" "A") ("A" "B" "C"))]))
 
(: alphabetize : Lines -> Lines)
(define (alphabetize all-shifts)
  (sort all-shifts shift<?))
 
(module+ test
  (check-equal?*
    [(alphabetize '(("A" "B" "C") ("B" "C") ("A")))
     '(("A") ("A" "B" "C") ("B" "C"))]))
 
; Lexicographic order on equal-length lists of words
(: shift<? : Words Words -> Boolean)
(define (shift<? shift1 shift2)
  (match* (shift1 shift2)
   [('() _) ; first list empty, don't care about second
    #t]
   [(_ '()) ; first list non-empty, second empty
    #f]
   [((cons s1 shift1-rest) (cons s2 shift2-rest))
    (or (string<? s1 s2)
        (and (string=? s1 s2)
             (shift<? shift1-rest shift2-rest)))]))
 
(module+ test
  (check-equal?*
    [(shift<? '() '())
     #t]
    [(shift<? '("A" "B") '("A" "C"))
     #t]))
 
(: kwic-display : Lines -> Void)
(define (kwic-display all-sorted-shifts)
  (: display-words : Words -> Void)
  (define (display-words words)
    (display (first words))
    (for ([word (in-list (cdr words))])
      (display " ")
      (display word))
    (newline))
  (for-each display-words all-sorted-shifts))
 
(module+ test
  (parameterize ([current-output-port (open-output-string)])
    (kwic-display '(("A") ("B" "C")))
    (check-equal?
      (get-output-string (current-output-port))
      "A\nB C\n")))
 
(: all-circular-shifts* : Lines -> (Listof Lines))
(define (all-circular-shifts* lines)
  (map all-circular-shifts lines))
 
(module+ test
  (check-equal?
    (all-circular-shifts* '(("A" "B" "C") ("D")))
    '((("C" "A" "B") ("B" "C" "A") ("A" "B" "C")) (("D")))))
 
(: kwic-index : Path-String -> Void)
(define (kwic-index file-name)
  (define all-lines (kwic-split (kwic-read file-name)))
  (define all-shifts (append* (all-circular-shifts* all-lines)))
  (kwic-display (alphabetize all-shifts)))
 
(module+ test
  (parameterize ([current-output-port (open-output-string)])
    (define tmpfile (make-temporary-file))
    (with-output-to-file tmpfile #:exists 'replace
      (λ ()
        (displayln "imagine if this")
        (displayln "took 2 weeks to write")))
    (kwic-index tmpfile)
    (delete-file tmpfile)
    (check-equal?
      (get-output-string (current-output-port))
      (string-join '(
        "2 weeks to write took"
        "if this imagine"
        "imagine if this"
        "this imagine if"
        "to write took 2 weeks"
        "took 2 weeks to write"
        "weeks to write took 2"
        "write took 2 weeks to\n") "\n"))))
 
(module+ main
  (require racket/cmdline)
  (: *output-to* (Parameterof Any))
  (define *output-to* (make-parameter #f))
  (command-line
    #:program "kwic index"
    #:once-each
    [("-o" "--output")
     output-to ; user-supplied input
     "Write output to file"
     (*output-to* output-to)] ; update the parameter *output-to*
    #:args (file-name)
    (define output-to (*output-to*)) ; read from parameter *output-to*
    (define out-port
      (if (string? output-to)
        (open-output-file output-to #:exists 'replace)
        (current-output-port)))
    (parameterize ([current-output-port out-port])
      (kwic-index (cast file-name Path-String)))
    (when (string? output-to)
      (close-output-port out-port))))
+ +

+ +
Sample interactions: +
+ +
+
+ + + + + + + + + + + + + + + +
> racket kwic.rkt
kwic index: expects 1 <file-name> on the command line, given 0 arguments
 
> echo "It is a truth universally acknowledged" > pride-and-prejudice.txt
> racket kwic.rkt -o index.out pride-and-prejudice.txt
> wc -l index.out
6 index.out
+ +

Closing

+ +

We started with functions, wrote (and quarantined) unit tests, + reinforced our design with types, and added a command-line interface. +Going forward we could add Scribble documentation and share our work as a package.

+ +

+ +
For more on building languages with Racket: +
+ +
+
+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/08/03/a-few-cores-too-many/index.html b/blog/2016/08/03/a-few-cores-too-many/index.html new file mode 100644 index 00000000..ee284d93 --- /dev/null +++ b/blog/2016/08/03/a-few-cores-too-many/index.html @@ -0,0 +1,215 @@ + + + + + + A few cores too many + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

A few cores too many

+

+ :: performance, benchmarking, lost time

+

By: Ben Greenman

+
+ +

Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.

+ + +

Last year, we proposed a method for evaluating the performance of gradual type systems based on measuring every possible configuration of typed and untyped code that a programmer might explore (pdf). Given the freedom that gradual typing offers, this is the only realistic way to measure the performance of a gradual type system.

+ +

But it is a lot to measure! While developing the method, we spent over 3 months benchmarking a total of 75,844 configurations. Each configuration is a complete program and some gradual typings caused some programs to slow by 50x or even 100x, so many of these configurations took minutes to run.

+ +

The next question we asked was naturally “how can we scale this method to large software projects?” In our case, the number of gradually typed configurations scaled exponentially with the number of modules. Current gradual type system for Python and JavaScript are exponential in the number of variables in the program.

+ +

We explored two solutions:

+ +
    +
  1. Max New began work on a prediction model (inspired by work on software product lines) to estimate the performance of 2^N configurations after polynomially-many measurements.
    2. +
  2. Asumu Takikawa and I shopped for a multi-core computer.
+ +

By Thanksgiving, we had bought a Linux machine with 2 AMD Opteron 6376 2.3GHz processors (16 cores each) and put it to work running benchmarks on 29 cores simultaneously. Life was good.

+ +

Later that winter, Max implemented a prediction algorithm. The basic idea was to focus on boundaries between modules and isolate their effect on performance. If two modules are untyped, their boundary will have zero cost. If the same two modules are typed, their boundary might result in an overall performance improvement due to type-driven optimizations. And if one module is typed and the other untyped, their boundary will suffer some cost of type checking at runtime. In general a program with N modules has at most N(N - 1) / 2 internal boundaries, so it is far more time-efficient to measure only the boundaries than to benchmark 2^N gradually typed configurations.

+ +

Fast-forward to March, we had a prototype prediction algorithm and it was time to test. Again using 29 cores (because, why not), we gathered cost/benefit numbers for one 4-module benchmark and used them to predict performance for its 16 configurations. The results were not very good.

+ +
Figure 1: True running time vs. predicted running time for 16 configurations +

Figure 1: True running time vs. predicted running time for 16 configurations

+ +

Those green circles are the ground truth, the average running time after 5 iterations of each config. The blue triangles are what we predicted. Except for configurations 0 and 8, the triangles are FAR off from the truth. Many are even negative … obviously the algorithm needs work.

+ +

But then, out of frustration, desperation, or just good luck, Max compared the predictions to ground truth data gathered on a single core, leaving the other 31 cores idle.

+ +
Figure 2: Predictions made using measurements from a single core +

Figure 2: Predictions made using measurements from a single core

+ +

First off, the red “sequential truth” dots are slightly closer to the predicted triangles. Second — and this is the scary part — the red dots are very different from the green dots. Running on 1 core vs. 29 cores should not change the measurements!

+ +

From here we tried increasing the running time of the benchmark, removing I/O and system calls, checking for hyperthreading (ARM cores don’t support it), and even changing the cores’ CPU governor. The hope was that results taken from 1 core could match results from N cores, for some N > 1. It turns out N = 2 was stable, but even for N = 3 we found graphs like the following:

+ +
Figure 3: exact running times. Same-colored dots in each column should be tightly clustered. +

Figure 3: exact running times. Same-colored dots in each column should be tightly clustered.

+ +

This data is for the same 16 configurations as the previous two graphs. Green dots are exact running times measured with 25 cores. Red dots are running times measured with 1 core. The red dots are much closer together, and always unimodal. The green dots are evidence that maybe the 32-core machine has, as Jan Vitek put it, 30 cores too many.

+ +
+

“Oh my. You think it’ll never happen to you. Well, now I’ve learned my lesson.”

+ +

And so, we said goodbye to the last 4 months of data and started over running at most two cores. The new results are all stable, but still we keep pinching ourselves.

+ +

P.S. the results from POPL 2016 are just fine, as they were not taken on the new machine running more than 2 cores. If you have time to confirm, that data is in our artifact and in the gradual-typing-performance repo.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/index.html b/blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/index.html new file mode 100644 index 00000000..514cf565 --- /dev/null +++ b/blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/index.html @@ -0,0 +1,199 @@ + + + + + + NEPLS on October 7th at Northeastern University + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

NEPLS on October 7th at Northeastern University

+

+ :: NEPLS, conference

+

By: Ben Greenman

+
+ +

The next edition of the New England Programming Language Seminar (NEPLS) will be held on Friday, October 7th at Northeastern University. Organizers are Gabriel Scherer and Max New. See you there!

+ + +

Here is the official announcement from the NEPLS mailing list.

+ +
+

Hi everyone,

+

The next New England Programming Languages and Systems Symposium will take place on

+
+

Friday, October 7th 2016

+

at

+
+

Northeastern University, Boston.

+

Please mark it in your calendars!

+

The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.

+

Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.

+

The submission deadline is

+
+

Sunday, September 25th.

+

Acceptance notifications will be out on Thursday, September 28th.

+

Send your proposal to

+
+

nepls-talks-committee@lists.cs.brown.edu

+

More details about NEPLS are available on the NEPLS webpage:

+
+

http://www.nepls.org/

+ +

To subscribe to the NEPLS mailing list, visit this page:

+ +

https://lists.cs.brown.edu/sympa/subscribe/nepls

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/10/11/compcert-overview/index.html b/blog/2016/10/11/compcert-overview/index.html new file mode 100644 index 00000000..7e15a9b2 --- /dev/null +++ b/blog/2016/10/11/compcert-overview/index.html @@ -0,0 +1,198 @@ + + + + + + CompCert Overview + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

CompCert Overview

+

+ :: tutorial, coq, compiler correctness

+

By: Ben Greenman

+
+ +

If you are interested in learning about the internals of the CompCert C compiler but would rather not read its source code, this post is for you.

+ + +

(This is a public service announcement.)

+ +

Last fall, I gave a short lecture on the 2006 paper “Formal Certification of a Compiler Back-End” by Xavier Leroy for Amal Ahmed’s “Special Topics in Programming Languages” class. Rather than present CompCert as it existed in 2006, I read the documentation and source code for CompCert 2.5 (released June 2015). The lecture then focused on three questions:

+ +
    +
  • What subset of C does CompCert handle, today?
    • +
  • What optimizing passes does CompCert perform?
    • +
  • What is the “correctness theorem” for CompCert, and what does this theorem mean?
+ +

My notes for the lecture give a “mid-level” summary of the compiler — there are more details than you’ll find in papers, but it’s (hopefully!) easier to read than the source code. The document is also hyperlinked to locations in the CompCert GitHub repository.

+ +

Here is the document:

+ +
+

http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf

+ +

And here is a table-of-contents:

+ +
    +
  1. Motivation, details of the source and target languages, high-level guarantees
    2. +
  2. Compiler pipeline, optimizing passes, links intermediate language grammars and Coq theorems
    3. +
  3. Background on compiler correctness
    4. +
  4. CompCert’s correctness, properties that CompCert does not guarantee
    5. +
  5. Recent (2006 – 2015) work in the CompCert ecosystem
+ +

The document ends with a short description of two other research projects that have grown into “industry software” and a link to Xaver Leroy’s OPLSS lectures on certified compilers. Enjoy!

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/index.html b/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/index.html new file mode 100644 index 00000000..700ddc1d --- /dev/null +++ b/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/index.html @@ -0,0 +1,223 @@ + + + + + + Emacs daemon for fast editor startup + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Emacs daemon for fast editor startup

+

+ :: System Administration, Emacs

+

By: Gabriel Scherer

+
+ +

In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs’ bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors — on each git commit for example, or when I use a Firefox extension to edit a textarea content in a proper editor.

+ +

In this blog post, I describe how to use emacsclient to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.

+ + +

Emacs has long supported a client/server mode: a daemon emacs instance is loaded in the background, and whenever you request a new emacs window you can creates a new frame (~ window) using the same instance. This means that the startup time is dramatically reduced after the daemon is launched, as for example the execution of your personal configuration code does not have to be repeated.

+ +

To use it, I have this code as /usr/bin/editor:

+ +
+ + + + +
+
+ 
1
+2
 +
+ 
#!/bin/bash
+emacsclient -a "" -c "$@"
+
+
+
+ +

The empty -a parameter means that if no daemon exists, it should start one in the background and retry. The -c option means that a new frame (window) should be created instead of reusing an existing one. "$@"means that when the script is invoked with a path as command-line parameter (editor /tmp/foo.txt), the corresponding file will be opened.

+ +

Finally, my .bash_profile sets the EDITOR variable to editor (export EDITOR=/usr/bin/editor); this environment variable is what most tools (git included) will use to invoke a text editor.

+ +

On my machine, starting the daemon takes 1.4s. Creating a client windows takes around 0.150s.

+ +

If you want to control the environment in which the daemon process is started, you can launch it explicitly by running emacs --daemon.

+ +

Cool kids use Spacemacs these days, which comes with all sort of convenient settings built in, and I’m told that it does daemonization out of the box. I haven’t taken the time to give Spacemacs a try yet.

+ +

Finally, sometimes having all editor windows share the same process is not the right thing, because I do stuff which makes Emacs a bit unstable. (It’s not very good at asynchronous communication with the rest of the world, so for example accessing a file through SSH from Emacs can hang the process when network goes bad.). I’ve been bitten a few times by a crash of all editor windows at the same time, and since then, when I know I’m going to do “heavy stuff”, I launch a separate process for it (just emacs instead of editor or emacsclient).

+ +

P.S.: To precisely measure the startup time, ask Emacs to evaluate a Lisp expression on startup, to kill it immediately.:

+ +
+ + + + +
+
+ 
1
+2
 +
+ 
$ time emacs --eval "(save-buffers-kill-terminal)"
+$ time emacsclient -a '' -c -e "(save-buffers-kill-terminal)"
+
+
+
+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/10/19/history-of-actors/index.html b/blog/2016/10/19/history-of-actors/index.html new file mode 100644 index 00000000..12aecaf4 --- /dev/null +++ b/blog/2016/10/19/history-of-actors/index.html @@ -0,0 +1,172 @@ + + + + + + History of Actors + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

History of Actors

+

+ :: history

+

By: Tony Garnock-Jones

+
+ +

Christos Dimoulas is currently teaching a “History of Programming Languages” class at Harvard. The class is, as Christos writes, “definitely not about this”; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.

+ +

On Monday, I presented “the History of Actors” for the class. I’ve made the written-out talk notes and an annotated bibliography available here.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/10/31/meaningful-distinctions/index.html b/blog/2016/10/31/meaningful-distinctions/index.html new file mode 100644 index 00000000..c767a8fa --- /dev/null +++ b/blog/2016/10/31/meaningful-distinctions/index.html @@ -0,0 +1,226 @@ + + + + + + Meaningful Distinctions + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Meaningful Distinctions

+

+ :: history, constructions

+

By: Ben Greenman

+
+ +
+

“Meaningful distinctions deserve to be maintained.” — Errett A. Bishop

+ +

Likewise, memorable quotations deserve to be read in context. In this spirit, I am happy to present the above “basic principle” in its context: Schizophrenia in contemporary mathematics (pdf)

+ +

Read on for a brief summary.

+ + +
+ +

I first read the above quotation in Michael Beeson’s introduction to the 2012 edition of Bishop’s Foundations of Constructive Analysis. That was two years ago.

+ +

Last month, I tried to find its context. Many other uses of the quote cited a Schizophrenia in comtemporary mathematics, but I could not find an electronic copy. (It turns out, the AMS Bookstore page for Erret Bishop: Reflections on Him and His Research includes a facsimile.)

+ +

Lest anyone else be tempted to conjure the ancient magic of inter-library loan, here is a scan of the pages I borrowed. Thanks to the University of Toledo for supplying the hard copy.

+ +
+

prl.ccs.neu.edu/img/sicm.pdf

+ +

The document is Bishop’s “feeling for the philosophical issues involved” in constructive mathematics. First, Bishop lists “schizophrenic attributes” (trouble spots) of contemporary mathematics. Next, he gives basic principles of constructivism and Brouwer’s interpretation of the logical quantifiers. Along the way, and as a source of examples, Bishop describes integers, sets, and real numbers. The emphasis is always on common-sense meaning and finite constructions.

+ +

After a brief summary and reflection, the last ten pages list recent advances in constructive mathematics and upcoming tasks. The open tasks are particularly interesting:

+ +
    +
  • systematically develop (constructive) algebra
    • +
  • give a constructive foundation for general topology
    • +
  • engage with the deeper “meaning of mathematics”
+ +

The popular quote on “Meaningful Distinctions” appears early in the paper, as one of Bishop’s four principles that “stand out as basic” to the philosophy of constructivism:

+ +
+

A. Mathematics is common sense.

+

B. Do not ask whether a statement is true until you know what it means.

+

C. A proof is any completely convincing argument.

+

D. Meaningful distinctions deserve to be maintained.

+ +

I had no idea that D was “a principle”, or that it had three siblings.

+ +

To further tempt you into reading the whole truth, here are some of my favorite phrases:

+ +
+
    +
  • One suspects that the majority of pure mathematicians … ignore as much content as they possibly can.
    • +
  • We have geared ourselves to producing research mathematicians who will begin to write papers as soon as possible. This anti-social and anti-intellectual process defeats even its own narrow ends.
    • +
  • … truth is not a source of trouble to the constructivist, because of his emphasis on meaning.
    • +
  • … guided primarily by considerations of content rather than elegance and formal attractiveness …
    • +
  • Let me tell you what a smart sequence will do.
    • +
  • Classical mathematics fails to observe meaningful distinctions having to do with integers.
    • +
  • Constructive mathematics does not postulate a pre-existent universe, with objects lying around waiting to be collected and grouped into sets, like shells on a beach.
    • +
  • It might be worthwhile to investigate the possibility that constructive mathematics would afford a solid philosophical basis for the theory of computation …
    • +
  • … if the product of two real numbers is 0, we are not entitled to conclude that one of them is 0.
    • +
  • It is fair to say that almost nobody finds his proof intelligible.
    • +
  • Mathematics is such a complicated activity that disagreements are bound to arise.
    • +
  • Algebraic topology, at least on the elementary level, should not be too difficult to constructivize.
    • +
  • I hope all this accords with your common sense, as it does with mine.
+ +

Now go find their context!

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/11/02/beta-reduction-part-1/index.html b/blog/2016/11/02/beta-reduction-part-1/index.html new file mode 100644 index 00000000..023d2366 --- /dev/null +++ b/blog/2016/11/02/beta-reduction-part-1/index.html @@ -0,0 +1,459 @@ + + + + + + Beta Reduction (Part 1) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Beta Reduction (Part 1)

+

+ :: lambda, calculus, beta, reduction, semantics

+

By: Milo Davis

+
+ +

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

+ + +

Introduction

+ +

This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn’t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.

+ +

This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in Racket, OCaml, and Haskell. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I’ll explain how to prove that β-reduction is confluent.

+ +

The λ-Calculus

+ +

The λ-calculus is a simple model of computation developed by Alonzo Church. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The BNF for the λ-calculus is as follows:

+ +
e ::= x
+   | λx.e
+   | e e
+ +

In the above BNF, x is a metavariable, standing for any variable. In this post, I use x, y, z, a, and b as variables in my examples. The λx.e term represents a function with a single parameter x. We say that the parameter, x is bound in e. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.

+ +

If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn’t want to program in this language as it’s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I’m not going to discuss these ideas in detail, but if you’re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in Python, Scheme, and JavaScript.

+ +

Now that we’ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I’m assuming the Barendregt variable convention (2.6) which states that every bound variable is distinct from every free variable.

+ +
x[ x := e ] = e
+y[ x := e ] = y
+(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ])
+(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])
+ +

With the substitution function defined, we can write a semantics for evaluation:

+ +
------------------------------
+  (λx.e1) e2 ->β e1[ x := e2 ]
+
+    e1 ->β e1'
+--------------------
+  e1 e2 ->β e1' e2
+
+    e2 ->β e2'
+--------------------
+  e1 e2 ->β e1 e2'
+ +

These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).

+ +

What is β-reduction?

+ +

More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function’s parameter in its body. More formally, we can define β-reduction as follows:

+ +
(λx.e1) e2 = e1[ x := e2 ]
+ +

Given that definition, lets look at a few examples. I’ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It’s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called “call by value”. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I’m going to ignore that here. This presentation of Haskell’s semantics is called “call by name” and the optimization is called “call by need”. (For more details on lazy evaluation, I recommend: Chang and Felleisen, ESOP 2012.)

+ +

Some Examples

+ +

The first example evaluates the same way in all of the languages.

+ +
    (λx.x) (λy.y)
+->β x[ x := (λy.y) ]
+=   (λy.y)
+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
    ((λ (x) x) (λ (y) y))
+->β x[ x := (λ (y) y) ]
+=   (λ (y) y)
+
+
+
+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
    (fun x -> x) (fun y -> y)
+->β x[x := (fun y -> y) ]
+=   x[x := (fun y -> y) ]
+
+
+
+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
    (\x -> x) (\y -> y)
+->β x[ x := (\y -> y) ]
+=   (\y -> y)
+
+
+
+ +

In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
    (\x -> \y -> y) ((\z -> z) (\a -> a))
+->β (\y -> y)[x := ((\z -> z) (\a -> a))]
+= (\y -> y)
+
+
+
+ +

Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
    ((λ (x) (λ (y) y)) ((λ (z) z) (λ (a) a)))
+->β ((λ (x) (λ (y) y)) (z[ z := (λ (a) a) ]))
+=   ((λ (x) (λ (y) y)) (λ (a) a))
+->β (λ (y) y)[ x := (λ (a) a) ]
+=   (λ (y) y)
+
+
+
+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
    (fun x -> (fun y -> y)) ((fun z -> z) (fun a -> a))
+->β (fun x -> (fun y -> y)) (z[ z := (fun a -> a) ])
+=   (fun x -> (fun y -> y)) (fun a -> a)
+->β (fun y -> y)[ x := (fun a -> a) ]
+=   (fun y -> y)
+
+
+
+ +

The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:

+ +
    (λx.λy.y) ((λz.z) (λa.a))
+->β (λx.λy.y) (z[ z := (λa.a) ])
+=   (λx.λy.y) (λa.a)
+->β (λy.y)[ x := (λa.a) ]
+=   (λy.y)
+ +
    (λx.λy.y) ((λz.z) (λa.a))
+->β (λy.y)[ x := ((λz.z) (λa.a)) ]
+=   (λy.y)
+ +

Lets look at a final example. This one is more complicated than the previous ones. I’m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what’s going on.

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
    ((λ (x) x) (((λ (a) (λ (b) (a b))) (λ (y) y)) (λ (z) z)))
+->β ((λ (x) x) ((λ (b) ((λ (y) y) b)) (λ (z) z)))
+->β ((λ (x) x) ((λ (y) y) (λ (z) z)))
+->β ((λ (x) x) (λ (z) z))
+->β (λ (z) z)
+
+
+
+ +

The same thing happens in OCaml

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
    (fun x -> x) ((fun a -> (fun b -> a b)) (fun y -> y) (fun z -> z));;
+->β (fun x -> x) ((fun b -> (fun y -> y) b) (fun z -> z));;
+->β (fun x -> x) ((fun y -> y) (fun z -> z))
+->β (fun x -> x) (fun z -> z)
+->β (fun z -> z)
+
+
+
+ +

In Haskell, the situation is a little different:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
    (\x -> x) ((\a -> \b -> a b) (\y -> y) (\z -> z))
+->β (\a -> \b -> a b) (\y -> y) (\z -> z)
+->β (\b ->  (\y -> y) b) (\z -> z)
+->β (\y -> y) (\z -> z)
+->β (\z -> z)
+
+
+
+ +

Finally, in the λ-calculus, things can go a few different ways.

+ +
    (λx.x) ((λa.λb.a b) (λy.y) (λz.z))
+->β (λx.x) ((λb.(λy.y) b) (λz.z))
+->β (λx.x) ((λy.y) (λz.z))
+->β (λy.y) (λz.z)
+->β (λz.z)
+ +
    (λx.x) ((λa.λb.a b) (λy.y) (λz.z))
+->β (λa.λb.a b) (λy.y) (λz.z)
+->β (λb.(λy.y) b) (λz.z)
+->β (λy.y) (λz.z)
+->β (λz.z)
+ +
    (λx.x) ((λa.λb.a b) (λy.y) (λz.z))
+->β (λx.x) ((λb.(λy.y) b) (λz.z))
+->β (λb.(λy.y) b) (λz.z)
+->β (λy.y) (λz.z)
+->β (λz.z)
+ +

There’s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:

+ +
(λx.λy.y) ((λa.a a) (λb.b b))
+ +

A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don’t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I’ll explain the proofs that the λ-calculus does indeed have these properties.

+ +

P.S. In Racket, you can look at the examples using the stepper which will allow you to interactively run the term and see how it reduces.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/11/16/understanding-constructive-galois-connections/index.html b/blog/2016/11/16/understanding-constructive-galois-connections/index.html new file mode 100644 index 00000000..29aee4f1 --- /dev/null +++ b/blog/2016/11/16/understanding-constructive-galois-connections/index.html @@ -0,0 +1,238 @@ + + + + + + Understanding Constructive Galois Connections + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Understanding Constructive Galois Connections

+

+ :: icfp, galois connection, adjunction, category theory, math

+

By: Max New

+
+ +

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

+ + +

History

+ +

I won’t go into much of the details of the paper, because I think it is quite well written, but here’s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of Abstract Interpretation. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the “on-paper” proofs, and offload much of the details to the proof assistant as mere calculation.

+ +

David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient St. Petersburg, Florida) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, “Kleisli” and “Constructive” Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.

+ +

Eventually, my usual trawling of Mathoverflow and nlab led me to a not-quite generalization to categories and interestingly a de-generalization to sets that helped me immensely to understand the theorem.

+ +

Since I know that the original theorem is a bit technical, I’ll explain the de-generalization to sets here, which I hope will help to understand their theorem.

+ +

Functions and Relations

+ +

Let’s start with the “Kleisli Arrows”, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).

+ +

Now to “de-posetize” this, we’ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).

+ +

Furthermore, the “Kleisli composition” is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]

+ +

Then the next thing we need to understand is what is the de-generalization of “Kleisli Galois connection”? Well, Galois connections are an instance of what’s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any “universe” that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I’m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and inclusion of relations, i.e. that one relation is a subset of another.

+ +

Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means

+ +

\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}

+ +

where \(\Delta \) means the diagonal, or equality relation on the set:

+ +

\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]

+ +

So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a “right-total” relation and \(Q \) is a “left-total” relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).

+ +

If we unwind (2), we get that for any \(y,y' \in Y\) if there’s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let’s see what this tells us about the relationship between \(R\) and \(Q \).

+ +

If \((x,y) \in R \), then by (1) there’s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we’ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we’ve discovered that actually \(Q \) is just the opposite relation of \(R \).

+ +

Then if we look at (2) again but replace the \(Q\)’s by flipped \(R\)’s we get that for any \(y,y' \in Y\), if there’s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).

+ +

You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!

+ +

Constructive Galois Connections

+ +

Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have “monotone relations”, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it’s the same as a Kleisli Galois connection and that a similar argument to the above shows that the “left adjoint” is represented by a monotone function \(f : A \to B \):

+ +

\[R = \{(x,y) | y \le f(x) \} \]

+ +

Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.

+ +

In fact, we get a little extra from what’s mentioned in their paper, which is that the “right adjoint” is represented by \(f \) as well but in the opposite way:

+ +

\[Q = \{(y,x) | f(x) \le y \}\]

+ +

Category Theory Post Scriptum

+ +

If you’re interested in Category theory, here’s a more technical addendum.

+ +

Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?

+ +

Well, first we need a generalization from relations to downward-closed relations to what are called distributors or profunctors. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?

+ +

The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) Cauchy completeness. Viewing sets and posets as special categories, it turns out that they always have this property, and that’s why the theorem worked out for those adjunctions.

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/11/17/src-submissions/index.html b/blog/2016/11/17/src-submissions/index.html new file mode 100644 index 00000000..eb6bc07f --- /dev/null +++ b/blog/2016/11/17/src-submissions/index.html @@ -0,0 +1,212 @@ + + + + + + SRC-submissions + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

SRC-submissions

+

+ ::

+

By: Gabriel Scherer

+
+ +

Max New, Daniel Patterson and Ben Greenman recently wrote three two-page abstracts on what they are working on right now. Come have a look — and any feedback is welcome!

+ + +

Gradual Type Precision as Retraction

+ +

Gradual Type Precision as Retraction +
Max New +
2016

+ +
+

Gradually typed programming languages allow for a mix of precision of static type information, allowing advanced type features to be added to existing languages, while still supporting interoperability with legacy code. The advantages of gradual typing are enticing to researchers and practitioners alike, but a general theory of gradually typed languages is only beginning to emerge after a decade of research.

+

It has long been noted that there is much similarity between work on contracts and gradual typing, and the use of retracts in domain theory which were used to relate models of untyped and typed lambda calculus in Scott(1976) and Scott(1980). Here we take this connection seriously and consider how judgments in modern gradually typed languages can be framed in terms of retractions. While retractions in programming languages were originally studied in terms of denotational semantics in domains, our presentation will use only the most basic elements of category theory: composition, identity and equality of terms, so our formulation is equally applicable to axiomatic or operational semantics.

+

In particular we propose a semantic criterion for the notion of precision of gradual types, a common judgment in gradually typed languages (sometimes called naive subtyping for historical reasons). We relate it to a previous definition from Wadler and Findler(2009) that defines type precision in terms of blame. We show that our definition decomposes in a similar way into “positive” and “negative” type precision, but without depending on a specific notion of blame in the language.

+ +

Linking Types: Specifying Safe Interoperability and Equivalences

+ +

Linking Types: Specifying Safe Interoperability and Equivalences +
Daniel Patterson +
2016

+ +
+

All programs written in high-level languages link with libraries written in lower-level languages, often to expose constructs, like threads, random numbers, or automatic serialization, that aren’t possible in the high-level language. This linking usually takes place after compiling both languages to a common language, possibly assembly. In this sense, reasoning about crosslanguage linking means reasoning about compilation.

+

While most languages include cross-language linking (FFI) mechanisms, they are ad-hoc and can easily break the semantic equivalences of the source language, making it hard for source programmers to reason about correctness of their programs and hard for compiler writers to reason about correctness of their optimizations.

+

In this work, I design and motivate linking types, a language-based mechanism for formally specifying safe linking with libraries utilizing features inexpressible in the source. Linking types allows programmers to reason about their programs in the presence of behavior inexpressible in their language, without dealing with the intricacies of either the compiler or the particular language they are linking with.

+ +

Pruning Contracts with Rosette

+ +

Pruning Contracts with Rosette +
Ben Greenman +
2016

+ +
+

Contracts are a pragmatic tool for managing software systems, but programs using contracts suffer runtime overhead. If this overhead becomes a performance bottleneck, programmers must manually edit or remove their contracts. This is no good. Rather, the contracts should identify their own inefficiencies and remove unnecessary dynamic checks. Implementing contracts with Rosette is a promising way to build such self-aware contracts.

+ +

While we’re at it let’s rant on SRCs

+ +

These abstracts are submitted at POPL’s “Student Research Competition”. You submit an abstract, and if you get accepted to that thing, you get a bit of travel support money, and you have to prepare a poster and present it at the conference.

+ +

I have a firm dislike for the Competition part of that concept: I think that people think of research too competitively already, and that we should have less of that, not more. (Having some is unfortunately unavoidable in scarce-resource situations.) I think that the process of awarding prizes to students with the “best poster” is dumb — and borderline ridiculous.

+ +

On the other hand, my experience seeing them writing these extended abstracts is that it’s a useful exercise for them, and produces nice result — short, readable introductions to their ideas. And Jennifer Paykin convincingly argues that although writing a poster is rather painful, actually presenting it during the conference is interesting and useful. In her words, “it’s worth it to get the experience of authentic and fruitful discussions”. Plus having posters in the corridors of one’s lab is very nice.

+ +

I think we could have “Student Research Sessions” or “Student Poster Sessions”, where students are encouraged to present their work, would write those nice extended abstracts and posters, interact with researchers at the conference, and get travel money, without the ranking and prize stuff. (I would still encourage students to participate to SRC today, it seems to be worth it.)

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/index.html b/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/index.html new file mode 100644 index 00000000..2105911f --- /dev/null +++ b/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/index.html @@ -0,0 +1,169 @@ + + + + + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Getting Started in Programming Languages (Cross-Post)

+

+ :: tutorial, semantics, books

+

By: Jonathan Schuster

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2016/12/17/measuring-the-submission-review-balance/index.html b/blog/2016/12/17/measuring-the-submission-review-balance/index.html new file mode 100644 index 00000000..37363201 --- /dev/null +++ b/blog/2016/12/17/measuring-the-submission-review-balance/index.html @@ -0,0 +1,232 @@ + + + + + + Measuring the submission/review balance + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Measuring the submission/review balance

+

+ ::

+

By: Gabriel Scherer

+
+ +

How do researchers know whether they are doing “enough” or “too many” reviews? A measurable goal is to be review-neutral: to have demanded, through our submissions, as many reviews as we have produced as reviewers.

+ + +

Reviewing is good

+ +

I like to review academic papers. It is a very rewarding activity in many different ways. One gets to serve the academic community, helping it function smoothly. One gets a chance at acquiring a much better understanding of someone else’s work than idle paper-skimming allows. One gets to send feedback to our colleagues and help them improve their work and its presentation — it is also an essential way in which we can participate to the formation of student researchers all over the world. Finally, doing reviews helped me develop the skill the judge someone else’s work and of forcing oneself to come up with a decisive opinion — it is surprisingly difficult and only comes with training.

+ +

Doing reviews is also fairly time-consuming. I noticed that the time I spend on each review is generally stable (excursions into previous or related work excluded): around one day and a half for conference reviews, and at least twice more for journal reviews — I’m sure other people have wildly different figures, but I would expect it to be a noticeable time commitment in any case. (Workshop reviews are much easier, at least for the formats I have seen of 2-page extended abstracts, I’d say one hour per review.)

+ +

How many reviews?

+ +

Because it is so time-consuming, deciding whether to say “yes” or “no” to invitations to review a new paper is not easy: in general I want to say “yes” (unless I can tell that I will not enjoy reading the paper at all), but it is not reasonable to say “yes” all the time, because I also need to spend time on other things. When should I say “no” because I have done “too many” reviews already?

+ +

We can count the number of reviews that we have done, and we can also estimate the number of reviews that we have demanded of others through our submissions. A natural goal for researchers is to produce at least as many reviews as they demand; if everyone reached this goal, the peer-review system would be at equilibrium without imposing too much of a workload on anyone.

+ +

To estimate the number of reviews a researcher demanded from their peers, you can sum, for each of their submissions to a peer-reviewed venue, the number of reviews that they received, divided by the total number of authors of the submissions.

+ +

\[ \sum_{p \in \mathtt{Submissions}} \frac{\mathtt{reviews}(p)}{\mathtt{authors}(p)} \]

+ +

Out of curiosity, I just measured this balance for myself: over my years doing research I have “demanded” 10 workshop reviews and 28.5 conference reviews, and “produced” 6 workshop reviews and 17 conference reviews. If you think that an article would interest me, you shouldn’t feel bad about asking me to review it, for now. (On the other hand, my balance this year is positive, so I wouldn’t feel to bad about refusing if I had to.)

+ +

Of course, a researcher’s balance is highly dependent on where they are in their academic career — maybe more so that on their personal choices. Students are supposed to submit articles, but are offered few opportunities for doing reviews. When they are invited to do reviews, it is often as sub-reviewer, one review at a time. More established researchers participate in program committees, where they have to do a fair amount of reviews at once — ten to twenty can be typical in Programming Languages conferences. This means that one naturally starts with a deficit of reviews, and that the opportunity to become balanced or positive only comes over the years.

+ +

(There is much more that could be said about the dynamics of the submission/review balance. I think the idea that a single person should be neutral should not be taken too seriously, because the dynamics are so complex. For example, some people stop doing reviews with a negative balance (students going to the industry for example), so long-time researchers necessarily have a very positive balance that may make short-time researchers balance considerations mostly irrelevant. Another thing is that there is no point doing more reviews than required by the submission flow, and that doing more reviews would build up more reviewing debt under this neutrality criterion — you can never have everyone positive.)

+ +

Quality

+ +

This is only a comment on the quantitative aspects of reviewing. Much more important is the qualitative part: are the reviews you receive and produce good reviews? (There is no objective definition of what a good review is; I like reviews that are constructive, help improve the work and its presentation, and catch mistakes.) For a given paper, one or a few very good reviews is more helpful than many bad reviews, so one should not compromise on the quality of one’s reviews in order to reach a quantitative goal.

+ +

Advice for students?

+ +

While proof-reading this post (thanks!), Ben asked some questions that may be of interest to others — mostly students, I suppose.

+ +
+

If I want to be review-neutral, but I have to accumulate a “review debt” before I can start reviewing, does this mean I should accept my first opportunity to review and every one that follows (until I’m neutral)?

+ +

The answer is of course “no”: one should never feel forced to accept reviews. On the other hand, I do think that it is worthwhile for PhD students to take advantage of the reviews they are offered, so “saying yes most of the time” sounds like a reasonable strategy to me — this is just a personal opinion. Some reasons:

+ +
    +
  • +

    Reviewing is hard and takes training, I think it is good to start practicing early. Students are in a good situation to exercise their reviewing skills at a fairly calm peace (you won’t get many reviews anyway), and with more time than more senior people.

    • +
  • +

    Student reviews are often done as sub-reviewer: someone does a review, but also asks for your opinion and includes your sub-review in their review. It is a low-pressure way to do your first reviews, and the ability to exchange opinions with the other reviewer and discuss both reviews is really helpful. Students can also ask for feedback on their reviews to their advisor, which is also very helpful.

    • +
  • +

    Reviewing teaches a few useful things about writing papers as well — it’s always easier to recognize the flaws in others’ work.

+ +

On the other hand, I think you should not accept reviews at times when you cannot invest enough work in the review, or when doing so would be detrimental to you — whether you are on a deadline, or under too much pressure, or have conflicting commitments, etc. This is more important than anything about a submission/review balance.

+ +
+

Do you have any ideas for how young researchers / new researchers can reduce their “review footprint”? For example, is it possible to volunteer for reviews?

+ +

Yes, you can volunteer for reviews by telling the colleagues in your lab that you would be interested in doing reviews and that they should consider giving you some.

+ +

(With the increased use of double-blind submission processes, it is becoming more difficult to pass conference reviews to external researchers. This means that students are relatively unlikely to receive review offers from outside their close colleagues.)

+ +

Besides doing more reviews, the two other factors one could in theory play with are: submitting less papers, and having more co-authors. I think there is something to be said for the first one: one reason to not submit unfinished, buggy or topically-inappropriate articles is that it has a review cost. The second factor should not be considered, I think: “did this person contribute to the work?” should weight infinitely more for co-authorship decisions.

+ +

Note: Another thing you can ask for is reading reviews other people received. I think that reading reviews is also very helpful for research beginners — whether reviews of one’s own work or someone else’s. In particular, I wouldn’t know how to write reviews if I hadn’t had the opportunity to read reviews before that. If someone you are close to receives reviews, you should consider asking them whether you could have a look.

+ +
+

Is being a student volunteer at a conference equal to “one review”?

+ +

I think it is a distinct form of academic service. I don’t know how to measure the “conference organization cost” we impose to our academic colleagues. (If there are around 500 attendants to a typical Programming Languages conference, it means that for every 500 conferences you attend you should organize one all by yourself.)

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/01/02/fall-2016-pl-junior-retrospective/index.html b/blog/2017/01/02/fall-2016-pl-junior-retrospective/index.html new file mode 100644 index 00000000..cad77459 --- /dev/null +++ b/blog/2017/01/02/fall-2016-pl-junior-retrospective/index.html @@ -0,0 +1,199 @@ + + + + + + Fall 2016 PL Junior Retrospective + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Fall 2016 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell

+
+ +

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

+ +
    +
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. +
  2. they represent a significant increase in the reasoning power of types over programs.
+ + +

There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by posts like these. So we followed the pi-for-all lecture series given by Stephanie Weirich at OPLSS, which focuses on implementing a simple dependently-typed programming language.

+ +

After the pi-for-all lectures, we read chapter two of Edwin Brady’s dissertation on implementing dependently typed languages. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.

+ +

Along the way, we became sidetracked by Girard’s paradox. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.

+ +

It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a paper that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.

+ +

The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.

+ +

With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.

+ +

This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.

+ +

Thoughts on the format

+ +

Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).

+ +

[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]

+ +

If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the ATTAPL chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there

+ +

For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/01/03/-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src/index.html b/blog/2017/01/03/-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src/index.html new file mode 100644 index 00000000..e5f29efa --- /dev/null +++ b/blog/2017/01/03/-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src/index.html @@ -0,0 +1,169 @@ + + + + + + [Toward Type-Preserving Compilation of Coq, at POPL17 SRC (cross-post)](https://williamjbowman.com/blog/2017/01/03/toward-type-preserving-compilation-of-coq-at-popl17-src/) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Toward Type-Preserving Compilation of Coq, at POPL17 SRC (cross-post)

+

+ ::

+

By: William J. Bowman

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/02/15/conversational-context-and-concurrency/index.html b/blog/2017/02/15/conversational-context-and-concurrency/index.html new file mode 100644 index 00000000..dc6f374f --- /dev/null +++ b/blog/2017/02/15/conversational-context-and-concurrency/index.html @@ -0,0 +1,197 @@ + + + + + + Conversational Context and Concurrency + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Conversational Context and Concurrency

+

+ :: HOPL

+

By: Tony Garnock-Jones

+
+ + +

When programs are written with concurrency in mind, the programmer reasons about the interactions between concurrent components or agents in the program. This includes exchange of information, as well as management of resources, handling of partial failure, collective decision-making and so on.

+ +

These components might be objects, or threads, or processes, or actors, or some more nebulous and loosely-defined concept; a group of callbacks, perhaps. The programmer has the notion of an agent in their mind, which translates into some representation of that agent in the program.

+ +

We think about the contexts (because there can be more than one) in which agents exist in two different ways. From each agent’s perspective, the important thing to think about is the boundary between the agent and everything else in the system. But from the system perspective, we often think about conversations between agents, whether it’s just two having an exchange, or a whole group collaborating on some task. Agents in a conversation play different roles, join and leave the group, and build shared conversational state.

+ +

In this talk, I used the idea of these conversational contexts as a lens through which to view the development of various metaphors and mechanisms of communication and coordination. I presented four computational models for concurrent interaction:

+ +
    +
  • monitors, and shared memory concurrency generally
    • +
  • the actor model
    • +
  • channel-based communication
    • +
  • tuplespaces
+ +

These aren’t full programming languages, but there are many programming models that build upon them. In some cases, development of these ideas has progressed all the way up to system models including user interaction and so forth.

+ +

The linked lecture notes include informal sketches of reduction semantics for each of the four models, plus a handful of small examples to give a feel for them.

+ +

Lecture Notes:

+ + + +

Discussion summary:

+ + +

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/02/15/introducing-hopl-2017/index.html b/blog/2017/02/15/introducing-hopl-2017/index.html new file mode 100644 index 00000000..09342dae --- /dev/null +++ b/blog/2017/02/15/introducing-hopl-2017/index.html @@ -0,0 +1,214 @@ + + + + + + Introducing HOPL 2017 + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Introducing HOPL 2017

+

+ :: HOPL

+

By: Ben Greenman

+
+ +

This semester at Northeastern, Matthias Felleisen is organizing the History of Programming Languages seminar. Look for posts tagged HOPL for updates from the lectures.

+ + +

Once every 6 to 8 years (i.e., once every batch of Ph.D. students?), Matthias Felleisen teaches History of Programming Languages. Nominally, the course is a seminar. But unlike a typical seminar course, weekly topics are not the technical details from a handful of papers. Rather:

+ +
+

The primary goal is to understand (some of) the discipline as it exists today and how some of its major themes evolved.

+ +
+

The secondary goal is to develop basic skills for understanding and describing research themes. Every student will learn to study a theme via a series of papers, prepare an annotated bibliography, and present the key steps in the evolution of the theme.

+ +

Themes is the operative word. To set the tone, this semester started with “themes that NUPRL faculty members have developed over the many decades of their careers.”

+ +
    +
  • Matthias, Full Abstraction: From PCF to SPCF
    • +
  • Jan Vitek, From Encapsulation to Ownership
    • +
  • Will Clinger, Garbage Collection vs. Manual Allocation
    • +
  • Olin Shivers, Higher-order Flow Analysis
    • +
  • Amal Ahmed, Logical Relations: Stepping Beyond Toy Languages
    • +
  • Matthias, Programming Languages and Calculi
    • +
  • Jan-Willem van de Meent, Rescoring Strategies for Probabilistic Programs
    • +
  • (upcoming) Mitch Wand, Analysis-Based Program Transformation
    • +
  • (upcoming) Frank Tip, Refactoring
+ +

At this point in the course, we are just starting with the student presentations. As these presentations happen, we plan to push updates to this blog. All presentation materials are in the course repository:

+ + + +

Speakers’ notes and annotated bibliographies are in top-level folders in the repo. Discussion summaries and “unofficial” notes are in the top-level lecture_notes/ folder.

+ +

The list of upcoming presentations is online (along with the papers each presentation is based on):

+ + + +

Blogs posts for each talk should appear 2 weeks after the talk happens.

+ +
+ +

Links to past editions of HOPL:

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/02/21/bullets-are-good-for-your-coq-proofs/index.html b/blog/2017/02/21/bullets-are-good-for-your-coq-proofs/index.html new file mode 100644 index 00000000..d1ecfd01 --- /dev/null +++ b/blog/2017/02/21/bullets-are-good-for-your-coq-proofs/index.html @@ -0,0 +1,291 @@ + + + + + + Bullets are good for your Coq proofs + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Bullets are good for your Coq proofs

+

+ :: coq

+

By: Gabriel Scherer

+
+ +

I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.

+ +

Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.

+ + +

Why bullets

+ +

While you are doing a proof, Coq shows a list of subgoals that have to be proved before the whole proof is complete. Most proof steps will operate on the current active subgoal, changing the hypotheses or the goal to prove, but some proof steps will split it into several subgoals (growing the total list of goals), or may terminate the proof of the current subgoal and show you the next active subgoal.

+ +

Before bullets, a typical proof script would contain the proofs of each subgoal, one after another.

+ +
induction foo. (* this creates many subgoal *)
+
+proof of first subgoal.
+
+proof of second subgoal.
+ +

There are many ways to structure this to make the structure more apparent: people would typically have a comment on each subgoal, or make disciplined use of indentation and blank lines. But, in my experience, a major problem with this style was maintainability in the face of changes to the definitions or parts of automation. It could be very hard of what was happening when a proof suddenly broke after a change before in the file:

+ +
    +
  • +

    If a proof step now proves less things, then what used to be the end of a subgoal may not be anymore. Coq would then start reading the proof of the next subgoal and try to apply it to the unfinished previous goals, generating very confusing errors (you believe you are in the second subgoal, but the context talks about a leaf case of the first goal).

    • +
  • +

    If a proof step now proves more things, it is also very bad! The next proof steps, meant for the first subgoal (for example), would then apply to the beginning of the second subgoal, and you get very confusing errors again.

+ +

What we need for robustness is a way to indicate our intent to Coq, when we think that a subgoal is finished and that a new subgoal starts, so that Coq can fail loudly at the moment where it notices that this intent does not match reality, instead of at an arbitrary later time.

+ +

(The S*Case tactics used in (older versions of) Software Foundations can solve this problem if used in a carefully, systematic way, and additionally provides naming. Alexandre Pilkiewicz implemented an even more powerful cases plugin. Bullets are available in standard Coq since 8.4 (released in 2012), and can be used with no effort.)

+ +

There is not much discussion of bullets around; see the documentation in the Coq manual. I learned a lot from Arthur Azevedo de Amorim’s Bullets.v file.

+ +

Finally, some people don’t use bullets, because they systematically use so much automation that they never see subgoals — each lemma has a one-line proof. This is also a valid style. (I have been going to Adam Chlipala’s Formal Reasoning about Programs 2017 class, where Adam ignores bullets because that is his usual style.) Because I am not crushy enough to do this from the start, my proofs tend to start with cases and subgoals, and then I refine them to add more automation for robustness. I found bullets very useful for the first step, and during the refinement process.

+ +

Bullets

+ +

Bullets are actually a combination of two features, braces { ... } and actual list bullets — -, +, *, or homogeneous repetitions of those, for example -- or ***.

+ +

Braces

+ +

The opening brace { focuses the proof on the current subgoal. If you finish the proof of the subgoal, the following subgoal will not become accessible automatically; you have to use the closing brace } first. (If you finish the goal earlier than you think, you get an error.) Conversely, } fails if the subgoal is not complete. (If you fail to finish, you get an error.)

+ +

The previous example can thus be written as follows, and will be more robust:

+ +
induction foo. (* this creates many subgoal *)
+{
+  proof of first subgoal.
+}
+{
+  proof of second subgoal.
+}
+ +

If you also want to make sure that an error occurs if the number of subgoals changes (for example if new constructors are added to the inductive type of foo), you can use an outer layer of braces:

+ +
{ induction foo. (* this creates many subgoal *)
+  {
+    proof of first subgoal.
+  }
+  {
+    proof of second subgoal.
+  }
+} (* would fail if a new subgoal appeared *)
+ +

List bullets

+ +

A bullet, for example --, also focuses on the next subgoal. The difference is that when the subgoal is finished, you do not have a closing construction, you must use the same bullet to move to the next subgoal. (Again, this fails if the first proof step changes to prove too much or too little.) With bullets you would write

+ +
induction foo. (* this creates many subgoal *)
++ proof of first subgoal.
++ proof of second subgoal.
+ +

Bullets can be nested, but you must use different bullets for the different nesting levels. For example, if this proof is only one subgoal of a larger proof, you can use:

+ +
- induction foo. (* this creates many subgoal *)
+  + proof of first subgoal.
+  + proof of second subgoal.
+- (* would fail if a new subgoal appeared *)
+  rest of the proof
+ +

The natural ordering of bullets, I think, is by increasing number of lines: -, + then * (and then multi-character bullets, I guess). You can also mix bullets with braces: the opening brace resets the bullet scope, any bullet can be used again with the subgoal.

+ +

This gives a large space of freedom in how you want to use these features. You can use only braces, only bullets, braces and only one level of bullets, etc. My own style evolved with experience using the feature, and I will present the current status below.

+ +

My current bullet style

+ +

When deciding how to use bullets, one distinguishes the commands that preserve the number of subgoals and those that may create new subgoals. I use some additional distinctions.

+ +

Some tactics, for example assert, create a number of subgoals that is statically known, always the same for the tactic. I then use braces around each sub-proof, except the last one, which I think of as the “rest” of the current proof.

+ +
assert foo as H.
+{ proof of foo. }
+rest of the proof using H:foo.
+ +

(If the proof of foo takes several lines, I two-indent them, with the braces alone on their lines.)

+ +

Most tactics create a dynamic number of subgoals, that depends on the specifics of the objects being operated on; this is the case of case, destruct, induction for example. In this case, I open a brace before the tactic, and use a bullet for each subgoal.

+ +
{ induction foo; simpl; auto.
+- proof of first remaining subgoal.
+- proof of second remaining subgoal.
+  rest of the proof of the second subgoal.
+}
+ +

(Notice that the subgoal-creating step is vertically aligned with the proof steps: I use both braces and bullets, but take only one indentation level each time.)

+ +

As an exception, I may omit the braces if we are at the toplevel of the proof (Proof .. Qed serve as braces).

+ +

Note that omitting the braces here and using different bullets when you nest is also just fine. In my experience it gives proofs that are a bit more pleasant to read but also a bit more cumbersome to edit and move around.

+ +

Finally, a not-uncommon mode of use of “dynamic” tactics in the sense above is to expect all the cases, except one, to be discharged by direct automation (for example they are all absurd except one). When it is my intent that all cases but one be discharged (and not a coincidence), I express it by not using braces (this command preserves the number of subgoals), but marking the remaining subgoal with a new bullet without increasing the indentation level.

+ +
{ induction foo.
+- first subgoal.
+- second subgoal.
+  case blah; discharge all sub-subgoals but one.
++ remaining sub-subgoal of the second subgoal.
+  finish the sub-subgoal.
+- third subgoal.
+}
+ +

(This is the only time where I use several bullet levels.)

+ +

If you are the kind of programmer that is passionate about indentation style, I should now have tricked you to use bullets to propose a different variant. Otherwise, please consider using bullets anyway, for example by following the style above, it will make your life easier in the face of changing developments.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/02/21/datalog-for-static-analysis/index.html b/blog/2017/02/21/datalog-for-static-analysis/index.html new file mode 100644 index 00000000..2dfcfb8d --- /dev/null +++ b/blog/2017/02/21/datalog-for-static-analysis/index.html @@ -0,0 +1,204 @@ + + + + + + Datalog for Static Analysis + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Datalog for Static Analysis

+

+ :: HOPL

+

By: Ben Greenman

+
+ + +

Datalog is an old DSL that frequently appears in work on static analysis. This edition of HOPL 2017 explores the origins of Datalog in general, its early use in program analysis, and why Datalog remains a useful tool.

+ +

Full notes:

+ + + +
+ +

Datalog as a language was introduced by 1978 (its semantic foundations date back to 1976). It is predicate logic as a database query language. The traditional view of a Datalog program is a time invariant transformation over the time varying data stored in an external database.

+ +

In the early 1990’s, Uwe Aβmann designed a graph rewriting systems (EARS) that could:

+ +
    +
  1. Uniformly express various problems in static analysis
    2. +
  2. Systematically derive efficient solutions to such problems.
+ +

(Prior work had derived the same solutions with ad-hoc methods.) Aβmann’s system is equivalent to Datalog.

+ +

In 1993, Reps used the + CORAL deductive database (an implementation of Datalog) to derive an on-demand (read: lazy) implementation of program slicing from a specification of the slicing problem.

+ +

Both Aβmann’s and Reps work appeared in 1994. This was the first time Datalog had been used to implement a static analysis.

+ +

Researchers continue to use Datalog because:

+ +
    +
  • predicate logic (specifically: Horn clauses without function symbols or negation) is useful for expressing recursive relations … and static analyses are all about recursive relations
    • +
  • the language separates specifications from their implementation
    • +
  • there are many techniques for efficiently serving a Datalog query
    • +
  • these techniques have been implemented in at least one commercial Datalog engine
+ +

For an excellent description of how Datalog can benefit static analysis, see the introduction to Rep’s paper.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/02/28/linear-types-for-low-level-languages/index.html b/blog/2017/02/28/linear-types-for-low-level-languages/index.html new file mode 100644 index 00000000..3f653de3 --- /dev/null +++ b/blog/2017/02/28/linear-types-for-low-level-languages/index.html @@ -0,0 +1,193 @@ + + + + + + Linear Types for Low-level Languages + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Linear Types for Low-level Languages

+

+ :: HOPL

+

By: Daniel Patterson

+
+ + +

In this talk, we covered early papers (primarily, by Girard, Lafont, and Abramsky) on linear logic and its reflections into computation. The goal was to understand why linearity is often turned to as a principled way to control resource usage, as shows up in a language like Rust. From the very beginning, researchers realized the implications for “low-level” languages - that linear resources would eliminate the need for garbage collection, allow in-place mutation, and enable safe parallel computation. However, pure implementations of linearity incur lots of copying, doing away with any efficiency gained, and we covered a survey of papers that attempted to reconcile this contradiction by weakening linearity in controlled ways.

+ +

Notes:

+ + + +
+ +

Just after the talk, over lunch, we had a lab discussion about the phrase “low level”. Here are some thoughts:

+ +
    +
  • the phrase is relative, both over time and depending on the programming task at hand
    • +
  • a “low level” task is “one that you shouldn’t need to worry about” while solving your current task
+ +

And here are some example “low-level” tasks:

+ +
    +
  • Time and space management is “low level” when designing a new algorithm (the first question is correctness)
    • +
  • Calling conventions and endian-ness (facets of the damn machine running the programs) are almost always low-level
    • +
  • Whether a given value is serializable is usually low-level
    • +
  • Possible side effects, thrown exceptions, and optional arguments can all be considered “low level” aspects of library functions. This is low-level in the sense that “I’d rather use a simpler type to think about this library”
    • +
  • Managing type annotations is a low-level detail in ML programs
+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/02/28/pliss-oregon-without-the-greek/index.html b/blog/2017/02/28/pliss-oregon-without-the-greek/index.html new file mode 100644 index 00000000..d4c7fa25 --- /dev/null +++ b/blog/2017/02/28/pliss-oregon-without-the-greek/index.html @@ -0,0 +1,198 @@ + + + + + + PLISS: Oregon without the greek + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

PLISS: Oregon without the greek

+

+ :: event, lectures, language implementation

+

By: Jan Vitek

+
+ +

What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or PLISS for short) has set out to answer.

+ + +

PLISS logo

+ +

The school will feature twelve speakers versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.

+ +

Lectures cover current research and future trends in programming language design and implementation, including:

+ +
    +
  • Writing Just-in-time Compilers with LLVM with Jan Vitek
    • +
  • Performance Evaluation and Benchmarking with Laurie Tratt
    • +
  • Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller
    • +
  • High-Performance Fully Concurrent Garbage Collection with Richard Jones
    • +
  • Compiling Dynamic Languages with David Edelsohn
    • +
  • Language-support for Distributed Datastores with Suresh Jagannathan
    • +
  • Testing Programming Language Implementations with Alastair Donaldson
    • +
  • Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might
    • +
  • The evolution of Scala with Martin Odersky
+ +

A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.

+ +

Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.

+ +

More information at:

+ +

https://pliss2017.github.io

+ +

(Oregon is a reference to the OPLSS, in which you may also be interested.)

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/03/03/plt-redex-mf-apply/index.html b/blog/2017/03/03/plt-redex-mf-apply/index.html new file mode 100644 index 00000000..eb80c60f --- /dev/null +++ b/blog/2017/03/03/plt-redex-mf-apply/index.html @@ -0,0 +1,515 @@ + + + + + + PLT Redex: mf-apply + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

PLT Redex: mf-apply

+

+ :: PLT Redex, package, lang-extension

+

By: Ben Greenman

+
+ +

The mf-apply keyword is for checked metafunction application in PLT Redex. In other words, (mf-apply f x) is just like (f x), but errors if f is not a previously-defined metafunction.

+ +

Also, consider applying to attend The Racket School of Semantics and Languages in Salt Lake City this summer: http://summer-school.racket-lang.org/2017/

+ + +

Metafunctions vs. List Patterns

+ +

Have you used PLT Redex? Good! Maybe this has happened to you:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
+24
+25
+26
+27
+28
+29
+30
+31
+32
+33
+34
+35
+36
+37
+38
+39
+40
+41
+42
+43
+44
+45
+46
+47
+48
+49
+50
+51
+52
+53
+54
+55
+56
+57
+58
+59
+60
+61
+62
+63
+64
+65
+66
+67
+68
+69
+70
 +
+ 
#lang racket
+(require redex)
+
+;; -----------------------------------------------------------------------------
+;; 1. You define a language
+(define-language STLC
+  [V ::= integer boolean C]
+  [C ::= (closure Λ ρ)]
+  [Λ ::= (λ (x : τ) M)]
+  [M ::= (M M) V Λ x]
+  [τ ::= Int Bool (τ → τ)]
+  [ρ ::= ((x V) ...)]
+  [Γ ::= ((x τ) ...)]
+  [x ::= variable-not-otherwise-mentioned]
+  #:binding-forms (λ (x : τ) M #:refers-to x))
+
+
+;; -----------------------------------------------------------------------------
+;; 2. You define a few metafunctions
+(define-metafunction STLC
+  closure->lam : C -> Λ
+  [(closure->lam (closure Λ ρ))
+   Λ])
+
+(define-metafunction STLC
+  closure->env : C -> ρ
+  [(closure->env (closure Λ ρ))
+   ρ])
+
+
+;; -----------------------------------------------------------------------------
+;; 3. You try defining a judgment form . . .
+(define-judgment-form STLC
+  #:mode (free-variables I O)
+  #:contract (free-variables M (x ...))
+  [
+   --- FVS-Var
+   (free-variables x (x))]
+  [
+   (free-variables M_0 (x_0 ...))
+   (free-variables M_1 (x_1 ...))
+   --- FVS-App
+   (free-variables (M_0 M_1) (x_0 ... x_1 ...))]
+  [
+   (where (λ (x_0 τ) M) Λ)
+   (free-variables M (x_1 ...))
+   (where (x_2 ...) ,(set-remove (term (x_1 ...)) (term x_0)))
+   --- FVS-Λ
+   (free-variables Λ (x_2 ...))]
+  [
+   --- FVS-Integer
+   (free-variables integer_0 ())]
+  [
+   --- FVS-Boolean
+   (free-variables boolean_0 ())]
+  [
+   (where Λ (closure->lam C))
+   (free-variables Λ (x_0 ...))
+   (where ((x_1 τ_1) ...) (closure-env C))
+   (where (x_2 ...) ,(set-subtract (term (x_0 ...)) (term (x_1 ...))))
+   --- FVS-Closure
+   (free-variables C (x_2 ...))])
+
+
+;; -----------------------------------------------------------------------------
+;; 4. You test the judgment, and it mysteriously fails
+(judgment-holds
+  (free-variables (closure (λ (x : Int) x) ())
+                  ()))
+;; ==> #f
+
+
+
+ +

WHAT HAPPENED??!

+ +

The problem is this line in the FVS-Closure rule:

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
   ....
+   (where ((x_1 τ_1) ...) (closure-env C))
+   ....
+
+
+
+ +

which checks that the list (closure-env C) (whose first element is the symbol closure-env and second element is the symbol C) matches the pattern ((x_1 τ_1) ...).

+ +

Right.

+ +

Of course you meant to apply the metafunction closure->env but made a typo. And since the syntax for metafunction application is the same as the syntax for building a list, Redex doesn’t report an error.

+ +

We can fix this code with the new mf-apply keyword (available on GitHub or in a snapshot build):

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
   ....
+   (where ((x_1 τ_1) ...) (mf-apply closure-env C))
+   ....
+
+
+
+ +

Running raco make now gives a compile-time error.

+ +
  term: expected a previously defined metafunction
+    at: closure-env
+    in: (mf-apply closure-env C)
+ +

But I still need to type mf-apply correctly!

+ +

Leif Andersen says:

+ +
+

I should point out that this has the issue of you still need to type mf-apply correctly. ;)

+ +

That is, if you accidentally write:

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
   ....
+   (where ((x_1 τ_1) ...) (mf-applu closure-env C))
+   ....
+
+
+
+ +

Then the code compiles, thinking you intend to match a list of three elements against the pattern.

+ +

Never fear, there are at least two solutions.

+ +

Solution 1: rename mf-apply

+ +

A simple fix is to rename the mf-apply keyword to something shorter (and harder to mis-type):

+ +
+ + + + +
+
+ 
1
+2
+3
+4
 +
+ 
#lang racket
+(require redex
+         (rename-in redex
+           [mf-apply MF]))
+
+
+
+ +

Solution 2: the mf-apply lang extension

+ +

A fancier solution is to install the mf-apply meta-language.

+ +
  $ raco pkg install mf-apply
+ +

This language updates the readtable to interpret S-expressions that begin with #{:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
 +
+ 
#lang mf-apply racket
+(require redex)
+
+(term #{f x ...})
+
+
+
+ +

as a metafunction application:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
 +
+ 
#lang mf-apply racket
+(require redex)
+
+(term (mf-apply f x ...))
+
+
+
+ +

You the programmer only needs to write the #{....} syntax.

+ +

Source code is on GitHub:

+ + + +

(It’s the simplest lang-extension I know of)

+ +

What is PLT Redex?

+ +

PLT Redex is a library for semantics engineering. One of my favorite Redex features is it implements capture-avoiding substitution and α-equivalence for any language with a #:binding-forms specification (such as STLC, above).

+ +

You can read more:

+ + + +

And if you act now, you can become a Redexan between July 10 and July 14 at the summer school in Salt Lake City, Utah:

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/03/10/type-inference-in-stack-based-programming-languages/index.html b/blog/2017/03/10/type-inference-in-stack-based-programming-languages/index.html new file mode 100644 index 00000000..d1347251 --- /dev/null +++ b/blog/2017/03/10/type-inference-in-stack-based-programming-languages/index.html @@ -0,0 +1,187 @@ + + + + + + Type Inference in Stack-Based Programming Languages + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Type Inference in Stack-Based Programming Languages

+

+ :: HOPL

+

By: Rob Kleffner

+
+ + +

Stack-based languages occupy a niche in today’s programming language environment. The predominant stack-based language in use by programmers is Forth, and is found mostly on embedded devices. These languages also find use as compile targets for more popular languages: the CIL and JVM are both stack-based. Less popular but highly interesting languages to mention include Joy and Factor, known for their emphasis on higher-order stack-based programming.

+ +

The majority of stack-based languages are not statically typed, and it would be a stretch to call Forth even dynamically typed. As such, developing large projects in Forth or Factor can require great discipline on the part of the programmer to avoid type errors.

+ +

In this talk, I presented the development of type inference for stack-based languages as a linear sequence, divided into two overarching segments:

+ +
    +
  • An algebraic system known as stack effects
    • +
  • Systems that can be encoded as nested pairs in standard functional programming languages
+ +

The thread of research on stack effects began with Jaanus Pöial in the early 1990’s, and is a formalization of a commenting style well-known in the Forth community. The nested tuple systems were first examined by Okasaki in 1993 in the context of Haskell, and were later applied to higher-order stack-based languages. At the end, I give some avenues for extending the research on these systems, and list some pitfalls to be avoided in further research.

+ +

Full notes (as PDF documents) — see the git repository for more documents:

+ + +

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/03/15/tracing-jits-for-dynamic-languages/index.html b/blog/2017/03/15/tracing-jits-for-dynamic-languages/index.html new file mode 100644 index 00000000..57417a19 --- /dev/null +++ b/blog/2017/03/15/tracing-jits-for-dynamic-languages/index.html @@ -0,0 +1,187 @@ + + + + + + Tracing JITs for Dynamic Languages + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Tracing JITs for Dynamic Languages

+

+ :: HOPL

+

By: Ming-Ho Yee

+
+ + +

Traditional JIT (just-in-time) compilers are method-based: they compile “hot” (i.e. frequently executed) methods to native code. An alternative is trace-based or tracing JITs, where the compilation unit is a (hot) sequence of instructions. Typically, such sequences of instructions correspond to loops, where programs spend most of their execution time.

+ +

Where did the idea of tracing come from? What was appealing about it? How was tracing adapted for JITs and dynamic languages? What happened to Mozilla’s TraceMonkey, which used to be part of Firefox? Do any JITs today use tracing?

+ +

In this talk, I trace tracing JITs from their origins to some of their recent developments. I cover five papers: the original tracing paper, an implementation of a tracing JIT for Java, the TraceMonkey JIT for JavaScript, PyPy’s “meta-level” tracing, and a specific class of optimizations for tracing JITs.

+ +

(The idea of using the phrase “trace tracing JITs” is from Matthias Felleisen.)

+ +

All materials can be found in the course repository:

+ + + +
+ +

If you liked this post, you may also be interested in on-stack replacement.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/03/24/-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness/index.html b/blog/2017/03/24/-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness/index.html new file mode 100644 index 00000000..d8daf1b5 --- /dev/null +++ b/blog/2017/03/24/-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness/index.html @@ -0,0 +1,169 @@ + + + + + + [What even is compiler correctness? (cross-post)](https://williamjbowman.com/blog/2017/03/24/what-even-is-compiler-correctness/) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

What even is compiler correctness? (cross-post)

+

+ ::

+

By: William J. Bowman

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/04/04/top-five-results-of-the-past-50-years-of-programming-languages-research/index.html b/blog/2017/04/04/top-five-results-of-the-past-50-years-of-programming-languages-research/index.html new file mode 100644 index 00000000..eba722af --- /dev/null +++ b/blog/2017/04/04/top-five-results-of-the-past-50-years-of-programming-languages-research/index.html @@ -0,0 +1,227 @@ + + + + + + Top Five Results of the Past 50 Years of Programming Languages Research + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Top Five Results of the Past 50 Years of Programming Languages Research

+

+ :: HOPL

+

By: Ben Greenman

+
+ +

Over the past 50 years, which result from programming languages research has had the greatest impact on working programmers?

+ + +

The center of the universe for a working programmer is the language (or languages) they use. Fundamental results in programming languages (PL) research can re-shape this universe.

+ +

In HOPL two weeks ago, Matthias claimed that type soundness is the most useful and influential result to flow from PL research to PL practice in the last 50 years.

+ +

But 50 years is a long time, and there are many serious contenders for the title of greatest PL result. Here are my (alphabetized) picks for the top five:

+ +

Abstraction

+ +
+

My goal in library design is this; I want to have a precise, elegant, re-usable abstraction —Conal Eliott, BayHac 2014 (00:01:55)

+ +

By abstraction, I mean anything whose interface is not just “read the implementation”. Could be a tuple, module, object, structure, semaphore, macro, etc. Even the memory hierarchy pyramid in your operating systems textbook is an abstraction. They are everywhere, and they are what separates computer science (it’s about ideas) from electrical engineering (it’s about transistors). Thank you Peter Landin and J.H. Morris.

+ +

Generational Garbage Collection

+ +

I don’t know much about garbage collection. I do know that I want it, and I’m pretty sure that I wouldn’t have it (outside of research languages) without generational garbage collection. Thank you David Ungar.

+ +

Generic Programming

+ +

a.k.a. the mainstream interpretations of parametric polymorphism

+ +

The thought of programming in Java 1.4 is terrifying. Thank you Jean-Yves Girard and John C. Reynolds and Gilad Bracha and Martin Odersky and David Stoutamire and Philip Wadler.

+ +

Modularization

+ +

How can humans understand large software systems? By organizing the systems into smaller components (modules, objects) with well-defined interfaces. It’s hard to imagine, but once upon a time the question of how to divide a system into modules was a new research problem. Thank you D.L. Parnas.

+ +

Type Soundness

+ +

Let me make two modest claims:

+ +
    +
  • Soundness (with respect to a dynamic semantics) is a desirable property for a static type system.
    • +
  • A large number (at least, thousands) of working programmers agree that programming in a language with a sound, static type system is “a good thing”.
+ +

Neither of these claims were true 50 years ago. They are definitely true today. And the slogan “well typed programs do not go wrong (up to a well-defined set of runtime errors)” has become the catchphrase of PL research. Thank you Robin Milner.

+ +

Honorable Mentions

+ + + +
+ +

If you liked this post, you may also be interested in:

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/index.html b/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/index.html new file mode 100644 index 00000000..d3f05a0a --- /dev/null +++ b/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/index.html @@ -0,0 +1,192 @@ + + + + + + Type-Directed Compilation, Parts I and II + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Type-Directed Compilation, Parts I and II

+

+ :: HOPL

+

By: Leif Andersen, William J. Bowman

+
+ + +

Part I: Type-Directed Compilation, by Leif Andersen.

+ +

In this talk we discuss the history of type directed compilation. We start with Xavier Leroy’s seminal paper: Unboxed Objects and Polymorphic Typing, continue to TIL (Typed Intermediate Language), and finish up with TAL (Typed Assembly Language). We talk about what it means for a compiler to be typed preserving, and give examples of optimizations that are enabled by types.

+ +

Discussion summary:

+ + + +

Part II: Dependent Type-Directed Compilation, by William J. Bowman

+ +

A certifying compiler is not verified, but it produces a proof of correctness for each binary. This proof can be independently checked to show that the binary was compiled correctly, removing the compiler from the trusted code base. Certifying compilation has its roots in preserving type-preserving compilation, and in particular in preserving dependent types. We start the history of dependent-type-preserving compilation with a compiler from C to Assembly. We’ll see a result showing that preserving dependent types isn’t possible, and then we’ll do it anyway.

+ +

Discussion summary:

+ + + +

Notes (to appear here, eventually):

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/04/20/refinement-types/index.html b/blog/2017/04/20/refinement-types/index.html new file mode 100644 index 00000000..d3ac48bf --- /dev/null +++ b/blog/2017/04/20/refinement-types/index.html @@ -0,0 +1,184 @@ + + + + + + Refinement Types + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Refinement Types

+

+ :: HOPL

+

By: Kevin Clancy

+
+ + +

Roughly, a refinement type system is an extra layer of precision, enforced through subtyping, added onto an existing type system. A base type is decomposed into a set of base refinements, each of which denotes a subset of the values belonging to the base type. A subtyping relation respecting set inclusion can then be defined among the refinements of the base type. These subtyping relations can be lifted onto a subtyping relation for compound types using a standard arrow subtyping rule.

+ +

Extra type-checking precision sounds great, but what in practical terms does this precision look like? Freeman and Pfenning’s ’92 paper Refinement Types for ML proposes extending ML’s type definition language with constructs for decomposing a discriminated union type into a lattice of subtypes. For example, it allows the decomposition of a list type into a lattice including base refinements for empty lists, non-empty lists, and singletons. Those with experience in functional programming will realize this alleviates the dreaded and inescapable “non-exhaustive pattern match” warning, which tends to crop up in situations where the programmer understands that an exhaustive pattern match is not necessary.

+ +

In the late 90’s Xi and Pfenning advanced the state of refinement types by introducing a dependent refinement type system, implemented as a tool called Dependent ML. Their approach identifies a base refinement using a tuple of terms drawn from some computationally tractable constraint language called an index language. A list datatype can then be refined with a term of the linear integer arithmetic index language, denoting the subset of all lists having a specific length. One list refinement is then considered a subtype of another when a constraint solver can prove their index terms equal. Vazou et. al.’s recent project Liquid Haskell is another dependent refinement type system which decides subtyping among base types by invoking an SMT solver under a context-dependent set of constraints. It differs significantly from Dependent ML in that it refines base types with certain well-behaved program terms rather than indices.

+ +
+ +

Resources:

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/04/25/prl-at-snapl-17/index.html b/blog/2017/04/25/prl-at-snapl-17/index.html new file mode 100644 index 00000000..884777f7 --- /dev/null +++ b/blog/2017/04/25/prl-at-snapl-17/index.html @@ -0,0 +1,206 @@ + + + + + + PRL at SNAPL'17 + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

PRL at SNAPL’17

+

+ ::

+

By: Gabriel Scherer

+
+ +

PRL recently produced three papers for the SNAPL conference.

+ + + + +

Linking Types for Multi-Language Software: Have Your Cake and Eat It Too

+ +

Daniel Patterson and Amal Ahmed, 2017

+ +
+

Software developers compose systems from components written in many different languages. A business-logic component may be written in Java or OCaml, a resource-intensive component in C or Rust, and a high-assurance component in Coq. In this multi-language world, program execution sends values from one linguistic context to another. This boundary-crossing exposes values to contexts with unforeseen behavior—that is, behavior that could not arise in the source language of the value. For example, a Rust function may end up being applied in an ML context that violates the memory usage policy enforced by Rust’s type system. This leads to the question of how developers ought to reason about code in such a multi-language world where behavior inexpressible in one language is easily realized in another.

+

This paper proposes the novel idea of linking types to address the problem of reasoning about single-language components in a multi-lingual setting. Specifically, linking types allow programmers to annotate where in a program they can link with components inexpressible in their unadulterated language. This enables developers to reason about (behavioral) equality using only their own language and the annotations, even though their code may be linked with code written in a language with more expressive power.

+ +

Search for Program Structure

+ +

Gabriel Scherer, 2017.

+ +
+

The community of programming language research loves the Curry-Howard correspondence between proofs and programs. Cut-elimination as computation, theorems for free, ‘call/cc’ as excluded middle, dependently typed languages as proof assistants, etc.

+

Yet we have, for all these years, missed an obvious observation: “the structure of programs corresponds to the structure of proof search”. For pure programs and intuitionistic logic, more is known about the latter than the former. We think we know what programs are, but logicians know better!

+

To motivate the study of proof search for program structure, we retrace recent research on applying the logical technique of focusing to study the canonical structure of simply-typed λ-terms. We then motivate the open problem of extending canonical forms to support richer type systems, such as polymorphism, by discussing a few enticing applications of more canonical program representations.

+ +

Migratory Typing: Ten Years Later

+ +

Sam Tobin-Hochstadt, Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Ben Greenman, Andrew M. Kent, Vincent St-Amour, T. Stephen Strickland and Asumu Takikawa, 2017.

+ +
+

In this day and age, many developers work on large, untyped code repositories. Even if they are the creators of the code, they notice that they have to figure out the equivalent of method signatures every time they work on old code. This step is time consuming and error prone.

+

Ten years ago, the two lead authors outlined a linguistic solution to this problem. Specifically they proposed the creation of typed twins for untyped programming languages so that developers could migrate scripts from the untyped world to a typed one in an incremental manner. Their programmatic paper also spelled out three guiding design principles concerning the acceptance of grown idioms, the soundness of mixed-typed programs, and the units of migration.

+

This paper revisits this idea of a migratory type system as implemented for Racket. It explains how the design principles have been used to produce the Typed Racket twin and presents an assessment of the project’s status, highlighting successes and failures.

+ +

.

+ +

SNAPL is not dissimilar to the (french-speaking) JFLA that I am more familiar with — with an added irritating call for paper and unreasonable registration price. It has an interesting diversity of topics of presentation: see also the complete list of accepted papers this year, and the list of the previous edition.

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/04/28/what-is-soft-typing/index.html b/blog/2017/04/28/what-is-soft-typing/index.html new file mode 100644 index 00000000..1ee011e8 --- /dev/null +++ b/blog/2017/04/28/what-is-soft-typing/index.html @@ -0,0 +1,268 @@ + + + + + + What is Soft Typing? + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

What is Soft Typing?

+

+ :: HOPL

+

By: Ben Greenman

+
+ + +

A soft type system rewrites programs and meets a few design criteria.

+ +
+ +

What are the Design Criteria?

+ +

According to Mike Fagan’s 1991 dissertation, a soft type system must:

+ +
    +
  • accept all syntactically correct programs as input;
    • +
  • produce equivalent, memory-safe programs as output; and
    • +
  • be unobtrusive
+ +

Important details:

+ +
    +
  • In this context, memory safe basically means “no segfaults”. Programs output by a soft type system should be as safe as statically-typed Java programs or dynamically-typed Python programs.
    • +
  • Fagan characterizes unobtrusive with two general principles:
    • +
  • minimal text principle : the type checker should work without any programmer-supplied annotations
    • +
  • minimal failure principle : the type checker should assign useful types to idiomatic programs (basically, don’t just say that every expression has “unknown” or “top” type)
+ +

Why would I want to use a soft type system?

+ +

If you:

+ +
    +
  • like dynamic typing
    • +
  • want some benefits of static typing
    • +
  • refuse to (or cannot!) change your code to satisfy a type checker
+ +

then Soft Typing is a perfect fit. You just need to find/build a soft type checker.

+ +

Clarification

+ +

The benefits of static typing that a soft type system can give are:

+ +
    +
  • early detection of typos and simple logical errors
    • +
  • documentation, through (inferred) type signatures
    • +
  • speed, when the types can justify removing a runtime safety check
+ +

See Andrew Wright’s 1994 dissertation for proof.

+ +

Can I use Andrew Wright’s soft type system?

+ +

Not sure, but you may download the code for it:

+ + + +

Please explain Fagan’s / Wright’s soft types

+ +

Types t are made of constructors k, flags f, and type variables a. The grammar for types is basically:

+ +
  t ::= a | (k f t ...) U t
+  k ::= Int | Pair | ->
+  f ::= ++ | -- | b
+  a ::= a0 | a1 | a2 | a3 | ....
+  b ::= b0 | b1 | b2 | b3 | ....
+ +

where:

+ +
    +
  • U is just a symbol, represents “union”
    • +
  • a is a type variable; there are infinitely many type variables
    • +
  • b is a flag variable; the set of flag variables is also infinte
+ +

There are also some rules for types to be well-formed.

+ +

Here are two well-formed types:

+ +
(Int ++) U a0
+
+(-> ++ ((Int b0) U a1)
+       ((Int ++) U a2)) U a3
+ +

Here are two types that match the grammar, but are NOT well-formed:

+ +
(Int ++ a0) U a1
+
+(-> --) U a2
+ +

Finally, some intuition:

+ +
    +
  • A constructor k is like a behavior,
    • +
  • a type describes the behaviors a value can have.
    • +
  • The description is like a bitvector of “yes”, “no”, or “maybe” for each possible behavior.
    • +
  • A flag variable is the way to say “maybe”.
    • +
  • Every type ends with a type variable because every typed expression might flow to a context that expects a more general type.
+ +

The type and flag variables let Fagan and Wright encode subtyping using polymorphism. It’s a very cool idea, introduced in Didier Remy’s POPL 1989 paper. But it adds a learning curve, and has some drawbacks for type inference.

+ +

Stream-of-consciousness notes from the HOPL lecture

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/index.html b/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/index.html new file mode 100644 index 00000000..5f0a2bc1 --- /dev/null +++ b/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/index.html @@ -0,0 +1,175 @@ + + + + + + Categorical Semantics for Dynamically Typed Programming Languages + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Categorical Semantics for Dynamically Typed Programming Languages

+

+ :: HOPL, category theory, dynamic typing, gradual typing

+

By: Max New

+
+ + +

In 1969, Dana Scott wrote an unpublished manuscript in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote a paper that organized many of the different semantics he and others had since found using the language of category theory.

+ +

This latter paper is really the first deserving of the title “categorical semantics of dynamic typing”, and so I’m going to present some of the theorems and “theorems” presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.

+ +

My Full Notes continue the story, and you might also be interested in the discussion during the lecture.

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/05/04/rank-polymorphism/index.html b/blog/2017/05/04/rank-polymorphism/index.html new file mode 100644 index 00000000..9b81bed8 --- /dev/null +++ b/blog/2017/05/04/rank-polymorphism/index.html @@ -0,0 +1,207 @@ + + + + + + Rank Polymorphism + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Rank Polymorphism

+

+ :: array language

+

By: Justin Slepak

+
+ +

Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let’s just call it lerp) for scalars:

+ +
(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))
+ +

The number marks on each argument indicate the expected “rank” of the argument: how many dimensions it should have. In this case, each one is marked 0, indicating a scalar (i.e., 0-dimensional) argument. The function is usable as-is for

+ +
    +
  • +

    α-blending two RGB pixels

    • +
  • +

    dimming or brightening an image

    • +
  • +

    fade transition between video scenes

+ + +

Each of these use cases mixes the argument dimensions a little differently. A pixel is a vector (a rank–1 structure) of numbers representing color channel values, so the α-blending case uses two vector arguments and one scalar argument.

+ +

The only real difference between these use cases is the iteration space: they’re all effectively loop nests around the same basic scalar operation. In a rank-polymorphic language, the iteration space is derived automatically from the data, so you don’t need to write out the control structure yourself.

+ +

The fundamental idea behind function application here is breaking the argument arrays into lower-ranked pieces called “cells.” Each cell has the rank expected by the function being applied. In the case of lerp, the pixels, images, videos, etc. are all broken up into rank–0 (scalar) cells because lerp expects rank–0 arguments. Other expected ranks are possible as well— a vector dot product function dot-prod would call for rank–1 cells, and a matrix inversion function minv would call for rank–2 cells.

+ +

The structure built up around the cells is called the “frame.” A matrix array is a rank–2 frame containing rank–0 cells for lerp, but it would be a rank–1 frame containing rank–1 cells for dot-prod and a rank–0 frame containing a single rank–1 cell for minv. A rank-n array could be broken down into a frame of cells in n+1 different ways, and it’s the function being applied that determines which decomposition to use.

+ +

Unfortunately, the implicit control structure that’s so convenient for the programmer is a problem for a compiler. Historically, implementations of such languages have had to do without static information about the iteration space. Interpreters (and line-at-a-time compilers, to a lesser extent) get to inspect the concrete data they’re dealing with, but static compilers have had to make do with emitting a generic loop structure. A “loop” over a scalar might sound like trivial overhead, but not when it appears within some other hot loop. Being unable to see when loop boundaries match up is also a barrier to loop fusion. The lack of thorough static shape information was a long-standing problem my advisor pointed out to me when I was a new student looking at possible research projects, and he was interested in applying some form of static analysis to gather that information.

+ +

The first step in addressing it was to come up with a formal semantics for rank polymorphism. Although APL has existed since the 1960s, it had mostly lived in a separate world from mainstream programming language research. The formal techniques developed in PL had seen little to no application to APL and its “successor” language J.

+ +

There’s a lot to dislike about APL and J—special case behavior in many of the primitive operators, limited function arity, syntactic separation of first-order and second-order functions, the impossibility of parsing an entire program at once (fun fact: static analysis has been tried there)—and of course the idiosyncratic identifiers used for primops have prompted plenty of internet arguments. None of those things are essential to the programming model, so I’m building a new language called Remora to isolate the aspects I want to study.

+ +

People don’t always think of a type system as a form of static analysis, but it turned out to be an effective way of gathering shape information. Remora’s type system uses a restricted form of dependent types, in the style of Dependent ML. An array type is indexed by the shape, the numeric sizes of the array’s individual dimensions. Index polymorphism (i.e., Π types) allows functions to work on varying cell shapes and even varying cell ranks (which is essential for primitives like append and reduce, which operate along the major axis of arrays, no matter their rank). Frame-rank polymorphism, which gives rise to the control structure, remains completely implicit, leaving it to be identified by the type rule for function application. As a nice bonus, type soundness rules out run-time errors arising from incompatible argument shapes.

+ +
+ +

If you liked this post, you may also be interested in:

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/index.html b/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/index.html new file mode 100644 index 00000000..19f74d99 --- /dev/null +++ b/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/index.html @@ -0,0 +1,185 @@ + + + + + + No Good Answers, Gradually Typed Object-Oriented Languages + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

No Good Answers, Gradually Typed Object-Oriented Languages

+

+ :: HOPL, Gradual Typing

+

By: Ben Chung

+
+ + +

Untyped code remains a real problem in practice, as a result of reduced performance and hindered readability. One approach to solve this problem is gradual typing.

+ +

Gradual typing puts the onus on the developer to add type annotations, statically checks whatever type annotations have been written, and dynamically ensures that untyped code does not violate those annotations. A number of approaches have been put forward to try to achieve these objectives while retaining efficiency, semantic meaning, and the ability to actually type untyped code.

+ +

I discussed three systems, all of which achieve the objective of typing untyped code in different ways, and all of which have different tradeoffs.

+ +

Unofficial Notes:

+ + + +

Code Examples:

+ + +

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/05/15/artifacts-for-semantics/index.html b/blog/2017/05/15/artifacts-for-semantics/index.html new file mode 100644 index 00000000..02183c64 --- /dev/null +++ b/blog/2017/05/15/artifacts-for-semantics/index.html @@ -0,0 +1,210 @@ + + + + + + Artifacts for Semantics + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Artifacts for Semantics

+

+ ::

+

By: Daniel Patterson

+
+ +

Gabriel Scherer and I recently wrote an artifact for a semantics paper on a typed assembly language interoperating with a high-level functional language.

+ + +

We wrote a interpreter, typechecker, and parser in OCaml, compiled it to Javascript using js_of_ocaml, and then put it on a webpage (with an editor with syntax highlighting and error reporting) that allows people to step through examples from the paper or write their own. (Feel free to start by playing a bit with our artifact).

+ +

This post will summarize the different parts to make it easier for others to repeat this process. We think it was a total success, and have gotten feedback that it makes understanding the (somewhat complex) language from the paper much easier. We argue that building such interpreters / typecheckers is easy enough that all papers should do this. Further, while our interpreter / typechecker is completely unverified, since we wrote it in OCaml, this approach should work equally well for semantics verified in Coq and then extracted to OCaml.

+ +
+ +

The paper in question, FunTAL: Reasonably Mixing a Functional Language with Assembly (to appear in PLDI17), presents a multi-language that incorporates a typed assembly language (TAL) and a simple functional language where each can be embedded within the other. The paper then develops a logical relation that can be used to reason about the equivalence of such mixed programs. For example in the paper we show an imperative register-based factorial and a functional factorial equivalent.

+ +

Both the static and dynamic semantics are relatively complex. The typed assembly has registers (which store word-sized values), a heap (which stores code-blocks and tuples), and a stack (not a call-stack, simply a place where word-sized values can be pushed and popped). Code-blocks have pre-conditions on the state of the registers and the stack, and allow the tail of the stack to be abstracted over polymorphically. This allows values to be protected on the stack before jumping to blocks that otherwise could change them. This is used, along with a novel notion of return markers, to ensure well-bracketing in the control flow of the typed assembly. The return markers indicate the location that points to the block that will eventually be returned to (assuming it doesn’t loop infinitely). At the top level, the return marker end indicates that, assuming it does not loop, eventually the program will stop, rather than returning somewhere else.

+ +

Understanding the dynamic semantics requires tracking how values flow through the registers, the heap, and the stack, and rather than a call-stack, the user has to track the control flow through the statically-enforced return markers. This allows a good deal of low-level control-flow while still ensuring that calls will eventually return to the right place. This well-bracketing is vital to be able to reason about “components” that eventually return a value of a particular type, a necessity when embedding these components in a typed high-level program! However, it does mean that understanding the static and dynamic semantics from a few rules alone is a tall order. Our functional language is more standard, though we use (iso)-recursive types to allow recursion, which can easily trip up people, especially when you don’t have a type-checker to catch typos!

+ +

For that reason, when working through examples for the paper I implemented a simple interpreter for the multi-language. I did this in OCaml, in the most straightforward way possible: by translating the definitions from the paper into type definitions (for F and for TAL), and the reduction relation into a “step” function that (assuming it wasn’t stuck or a value), did one step of evaluation. Later, I did the same thing for the type-checker, translating rules into a type-checking function. The latter had to deviate from the rules in the paper in a few minor ways, as the typing rules we had in the paper were slightly not syntax directed.

+ +

Having the interpreter and type-checker was very useful for me, as I could check that the examples from the paper did not contain typos, but it was much less useful as an artifact for a reader of the paper. To use it the reader would have to download the source, install OCaml, write out examples as OCaml data constructors in a test file, compile it, run it, and then interpret the (quite overwhelming) output of every step of evaluation. At each step, I printed the current registers, current stack, current heap, what the evaluation context was (as you might be evaluating TAL instructions that were embedded inside a functional program that, in turn, was embedded in further TAL instructions), and what the current reduction was.

+ +

To get from that useful-for-the-authors artifact to a useful-to-readers artifact requires doing three things:

+ +
    +
  1. Allow reading/writing programs in a notation as close to the paper as possible. In our paper we use superscripts, subscripts, and a few greek letters, but ended up with a syntax otherwise very close to the paper — the biggest differences were a few extra delimiters introduced to reduce ambiguity.
    2. +
  2. Present an interface that highlights type errors at the location they occurred in, and allow a reader to step forward and backwards through the evaluation. Printing console output traces is fine for authors, but adds too much effort for readers.
    3. +
  3. Put it online! Don’t require installing any software! Conveniently, implementing 2 is also made easier once done online, as we could use existing editor tooling to present the code, highlight errors, etc. By using OCaml, we were able to easily use the excellent js_of_ocaml.
+ +

The first was done by Gabriel, who wrote a grammar using Menhir, and then equipped it with custom parsing error messages that provide much better feedback when there are typos in what people are trying. We also wrote a pretty-printer using the PPrint library, so we could show intermediate program states through the UI. After writing this, we were able to convert our existing suite of test cases and examples to be written textually, which was a huge improvement for us as well! These and other tests were used to ensure that the parser/pretty-printer would round-trip properly.

+ +

For the interface, I built a simple web page that had the CodeMirror editor equipped with a very simple syntax highlighter (8 lines of code to highlight keywords & atoms, plus a CodeMirror extension to highlight matching brackets) and error highlighting (which is triggered by the OCaml code). I then made a simple “machine state” UI that showed, in pretty-printed format, the heap, stack, registers, context, and redex. On the OCaml side, when the “run” button is clicked, we parse and typecheck and, assuming no errors occur, store the current state as our “history”. As the user clicks forward or backwards, we run the step function and append to the history of states or pop states off of the history. In total, there are 50 lines of Javascript and about 150 lines of OCaml that handle the logic for this interactive UI.

+ +

Putting it online was very easy, based on the choice of tools used earlier. We compile the main file (web.ml) to Javascript using js_of_ocaml, and it pulls in the parser, type-checker, interpreter, examples, etc. The rest of the artifact is a single html file, a CSS file, and a few javascript files for CodeMirror. It requires no server backend, is easy to archive and save, and will even run on smartphones!

+ +

The total time spent implementing the artifact was a small fraction of the time spent on the paper (probably 15 days of person-time), and while it was not in any critical way essential for the success of the paper, it does make the paper much easier to read, and we would argue that all semantics papers would be better off with easy to use artifacts for experimentation. Also, while implementing the artifact we found a few mistakes in the typing judgments for the language. The most interesting one was for our protect TAL instruction, which exists to protect the tail of the stack in a fresh type variable. We had written this as a typing rule that type-checked the rest of the instruction sequence with the abstracted tail, but this never allowed the tail to be accessed again. By translating the typing judgments exactly into code, we realized that there was a problem, because examples that should have worked did not type-check! We were then able to fix the typing rule to conform to what we originally thought it achieved — locally abstracting, but not abstracting from outside the component. What is interesting is that this did not come up in our proofs because the typing rule was perfectly valid — it just did not allow non-trivial programs that used the protect instruction. It’s quite possible we would have noticed this without implementing the artifact, but the artifact certainly made it easier!

+ +

To see the artifact online, visit:

+ +

https://dbp.io/artifacts/funtal

+ +

The source code is at:

+ +

https://github.com/dbp/funtal

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/05/23/building-a-website-with-scribble/index.html b/blog/2017/05/23/building-a-website-with-scribble/index.html new file mode 100644 index 00000000..61540562 --- /dev/null +++ b/blog/2017/05/23/building-a-website-with-scribble/index.html @@ -0,0 +1,574 @@ + + + + + + Building a Website with Scribble + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Building a Website with Scribble

+

+ :: Scribble, tutorial

+

By: Ben Greenman

+
+ +

The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!

+ + +

The Story so Far

+ +

Last Fall, I took a flight to Chicago (on my way to RacketCon 2016). When I landed, there was a new message in my inbox:

+ +
    Subject: Web Page
+    Date: 2016-09-15
+
+    You have been nominated webmaster by public acclamation. Congratulations!
+ +

Emboldened by the trust of my people, I promptly converted the PRL website from Racket-generating-HTML to the fine scribble/html preprocessor language (commit a0600d) This bold action polarized the community.

+ +
+

I can’t read the source anymore! Is this really an improvement?

+ +

Fear not, citizens. The switch to scribble/html was the right choice, and you too can learn to read the source code.

+ +

How to Read scribble/html Programs

+ +

Basics

+ +

Scribble is a language for writing Racket documentation. The key innovation in Scribble is the @-expression (read: “at expression”). The scribble/html language combines @-expression syntax with functions that generate HTML.

+ +

@-syntax

+ +

Greg Hendershott and the Scribble Documentation explain @-expressions properly. Here’s a short tutorial (Part 1 of 2, “the basics”):

+ +
    +
  • Scribble programs start in “text mode”. Every character you type goes straight to the document you are building.
    • +
  • The @-sign toggles to “Racket mode” for the next expression. In Racket mode, the characters you type will be evaluated as a Racket program to produce part of the document.
+ +

Examples: Evaluating "Hello Dave" puts “Hello Dave” in your document. Evaluating "Hello @Dave" puts “Hello ???” in your document, where "???" is the value of the variable Dave. Finally if Dave is the name of a function, then "Hello @(Dave)" calls the Dave function with zero arguments and puts whatever it returns into your document.

+ +

To make it easy to interleave text, function calls, and code, Scribble discriminates between 4 kinds of parentheses when they follow an @-sign (Part 2 of 2, “the parens”):

+ +
    +
  • @(f A B) is just like the function call (f A B) in Racket
    • +
  • @f[A B] is the same as @(f A B), but typically more useful because …
    • +
  • @f[A B]{....} evaluates the .... in “text mode” to a list of words w*, then calls f just like (apply f A B w*)
    • +
  • @f{....} evaluates the .... in “text mode” and calls f with the results
    • +
  • @f|{....}| is similar, but the .... are in “unescapable text mode”
+ +

“Unescapable text mode” treats @-signs as text instead of toggling between modes.

+ +

Generating HTML

+ +

The scribble/html language comes with functions that render HTML. These functions have the same name as the corresponding HTML tag.

+ +

Example program:

+ +
+ + + + +
+
+ 
1
+2
 +
+ 
#lang scribble/html
+@p{Hello World}
+
+
+
+ +

Running this program prints:

+ +
+ + + + +
+
+ 
1
 +
+ 
<p>Hello World</p>
+
+
+
+ +

No surprises.

+ +

One thing that is surprising is how scribble/html handles tag attributes. Every tag-rendering function accepts “Racket mode” arguments that specify an attribute name and attribute value.

+ +

For example:

+ +
+ + + + +
+
+ 
1
+2
 +
+ 
#lang scribble/html
+@p[style: "color:red"]{Hello World}
+
+
+
+ +

Prints:

+ +
+ + + + +
+
+ 
1
 +
+ 
<p style="color:red">Hello World</p>
+
+
+
+ +

Hope the output looks familiar. The input syntax is strange, but that’s what it is.

+ +

Larger programs print larger webpages. Each page on the PRL website is HTML generated by one scribble/html program.

+ +

Why scribble/html is an Improvement

+ +

Before scribble/html, the PRL website was implemented in scribble/text. A scribble/text program renders and prints text. There is no extra support for HTML.

+ +

To compare, here’s the start of the old homepage:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
+8
+9
 +
+ 
#lang scribble/text
+@(require "templates.rkt")
+
+<!DOCTYPE html>
+<html lang="en">
+  @(header "Home")
+  <body id="pn-top">
+    @(navbar "Home")
+    <div class="jumbotron">
+
+
+
+ +

And here is the start of the scribble/html’d homepage:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
+8
+9
 +
+ 
#lang scribble/html
+@require["templates.rkt"]
+
+@doctype{html}
+@html[lang: "en"]{
+  @header{Home}
+    @body[id: "pn-top"]{
+      @navbar{Home}
+      @div[class: "jumbotron"]{
+
+
+
+ +

The pages look similar. The new one has more @-signs and parentheses, the old one has more <-signs and quotes. If you were able to edit the old page, you should be able to edit the new page.

+ +

The key improvement in the new page is that common mistakes are now compile-time errors.

+ +
    +
  • +

    Before, a typo like <hmtl> would generate an ugly webpage. After, a typo like @hmtl is a syntax error.

    • +
  • +

    Before, a typo like <b>.... with no closing tag would generate an ugly webpage. After, a typo like @b{.... is a syntax error.

+ +

Both flavors of error message come with source-code line numbers. This is very very helpful.

+ +

Small Improvements

+ +

1. More Functions

+ +

Before, the Teaching page contained some interesting HTML for rendering vertical text (look for the word “Semantics” to see how this was used):

+ +
+ + + + +
+
+ 
1
 +
+ 
<span class="how-to-design-programs">S<br />e<br />m<br />a<br />n<br />t<br />i<br />c<br />s<br /><br /></span>
+
+
+
+ +

After, the same text is generated from a function call:

+ +
+ + + + +
+
+ 
1
 +
+ 
@span[class: "how-to-design-programs"]{@vertical-text{Semantics}}
+
+
+
+ +

The vertical-text function is simple:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
 +
+ 
@require[(only-in racket/list add-between)]
+
+@(define (vertical-text . str*)
+   (add-between (string->list (append* str*)) (br)))
+
+
+
+ +

2. More Structure, Less Boilerplate

+ +

Here’s part of the old definition of “Ben Greenman” on the People page:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
 +
+ 
<div class="row pn-person">
+  <div class="col-md-12 pn-row-eq-height">
+    <div class="col-md-3 pn-photo">
+      <div class="img-wrapper">
+        <img src="img/ben_greenman.jpg" title="Ben Greenman" alt="Ben Greenman" />
+      </div>
+    </div>
+    <div class="col-md-9">
+      <div class="col-md-4 pn-contact">
+        <span class="pn-name">Ben Greenman</span><br />
+        Advisor: Matthias Felleisen<br />
+        <a href="mailto:types@"@"ccs.neu.edu">types@"@"ccs.neu.edu</a><br />
+        <a href="http://www.ccs.neu.edu/home/types">www.ccs.neu.edu/home/types</a>
+      </div>
+      <div class="col-md-3 pn-muted col-md-offset-5">
+        Joined 2014
+      </div>
+      <div class="col-md-12 pn-bio">
+        <p>I like constructions .... </p>
+      </div>
+    </div>
+  </div>
+</div>
+
+
+
+ +

The new definition uses a helper function with keyword arguments for each “field” of the person:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
+8
 +
+ 
@person[#:name "Ben Greenman"
+        #:title "Advisor: Matthias Felleisen"
+        #:e-mail "types@ccs.neu.edu"
+        #:website "http://ccs.neu.edu/home/types"
+        #:history @list["Joined 2014"]
+        #:img "ben_greenman.jpg"]{
+  I like constructions ....
+}
+
+
+
+ +

3. Less String-Formatting

+ +

Before, the code did a lot of string formatting (link):

+ +
+ + + + +
+
+ 
1
+2
 +
+ 
(define (link url body)
+  (string-append "<a href=\"" url "\">" body "</a>"))
+
+
+
+ +

The new code has no need for such helper functions.

+ +
+ + + + +
+
+ 
1
 +
+ 
@a[href: url body]
+
+
+
+ +

Bottom Line

+ +

Scribble is a good language for making static HTML pages.

+ +
+ +

If you liked this post, you may also be interested in:

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/05/24/programming-language-conference-in-russia/index.html b/blog/2017/05/24/programming-language-conference-in-russia/index.html new file mode 100644 index 00000000..936c21da --- /dev/null +++ b/blog/2017/05/24/programming-language-conference-in-russia/index.html @@ -0,0 +1,207 @@ + + + + + + Programming Language Conference in Russia + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Programming Language Conference in Russia

+

+ ::

+

By: Artem Pelenitsyn

+
+ +

In April 3—5 I took part into a Russian conference exclusively devoted to programming languages: Programming Languages and Compilers (Google.Translated version of the site). I was a member of organizing committee and had a paper there.

+ +

This is the first conference in Russia highly focused on our area of PL. At least for the last several decades (I believe, there were conferences of the kind back in USSR). The conference was devoted to the memory of prominent Soviet PL-researcher from Rostov-on-Don, Adolf Fuksman who worked on ideas quite similar to what we know as the aspect-oriented programming back in the 70-s.

+ + +

We designed and implemented the conference with my colleagues from I.I.Vorovich institute of Mathematics, Mechanics and Computer Science, Southern Federal University (Rostov-on-Don, Russia). We aimed at gathering as much PL-researchers and enthusiasts from academia in Russia as we could. One of the consequences of the aim was a decision to run the conference in Russian. Though we missed expertise from our non-Russian speaking colleagues, we got thorough participation from all over Russia:

+ +
+

Saint-Petersburg, Moscow, Novosibirsk, Krasnoyarsk, Ekaterinburg, Kazan, etc.

+ +

I only mention here the cities with more than 1 mil. population in decreasing in number of conference participants order (and excluding Rostov itself, of course).

+ +

I particularly liked talks by invited speakers. When searching for ones, we targeted Russians who work at prominent universities and/or have some visibility at the international level. We ended up with two researchers: Ilya Sergey (University College of London) and Ekaterina Komendantskaya (Heriot-Watt U., Edinburg, UK). Interestingly, their talks were quite close to each other:

+ +
    +
  • I. Sergey, Dependent Types for verification of real-world programs,
    • +
  • E. Komendantskaya, Automated Theorem Proving for Type Inference, Constructively.
+ +

Both of them talked about types as a logic tool to ensure program correctness.

+ +

Biggest opening in this conference for me was a team from Languages Tools Laboratory of JetBrains. Surely, you heard about JB: either about their famous IDE, IntelliJ IDEA, or the Kotlin programming language (which, by the way, is endorsed for Android development these days). You may also have noticed that JB become sponsors of ECOOP and OPLSS this year. So we had a whole team of researchers from Saint-Petersburg office of JB. Among their topics: OCaml embedding of miniKanren (some results was presented on ML Workshop 2016), parser combinator libraries for OCaml and constrained graph querying (this is not specifically a PL problem, see arXiv:1502.02242 for details).

+ +

Otherwise the spectrum of topics presented on the conference was quite broad, here are some:

+ +
    +
  • Static analysis with PVS-studio (a sponsor talk),
    • +
  • supercompilation (a talk by researchers from Pereslavl-Zalesskiy, where the topic is actively developed for decades),
    • +
  • C++ and beyond (by a member of the ISO C++ committee),
    • +
  • architecture-agnostic parallel programming languages and compilation techniques for parallel architectures,
    • +
  • game semantics and ontologies for PL semantics,
    • +
  • program analysis,
    • +
  • compiler architectures.
+ +

Full program with links to slides in Russian is available here.

+ +

Let me mention my submission: that was a joint work with a student of mine on exploring design space for parser combinator libraries using programming language with direct support of effect system, namely Frank. Julia Belyakova also participated in the conference with her work on Coq-certified interpreter for an extension of lambda-calculus with concept-parameters (module-like kind of thing). The follow-up of that work is accepted for FTfJP workshop this year. You can also listen to her on the topic at the NEPLS next week.

+ +

I hope that we will find sources, time, and, most important, high quality submissions for PLC–2018.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/index.html b/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/index.html new file mode 100644 index 00000000..9a4c9705 --- /dev/null +++ b/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/index.html @@ -0,0 +1,190 @@ + + + + + + Racket 6.9 and Windows 10 Creators Update + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Racket 6.9 and Windows 10 Creators Update

+

+ :: racket, windows, bugs

+

By: Leif Andersen

+
+ +

Racket 6.9 was released in April and it has been smooth sailing for many people. However, some people using the Windows 10 Creators Update have been experiencing crashes, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.

+ +

The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.

+ +

A VirtualBox image with Racket preinstalled can be downloaded here:

+ + + +

The username and password for this machine are both racket.

+ + +
    +
  1. +

    The first thing you need to install is virtualization software. In principle it doesn’t matter what you install, but for this tutorial, we will use VirtualBox. Go to their downloads page and download and install the version for your platform (most likely Windows).

    2. +
  2. +

    Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which you can download here. The rest of this tutorial will assume you are using this image.

    3. +
  3. +

    Start up VirtualBox and import the virtual machine. You can do this by clicking on File -> Import Appliance. In the dialog, select the image you downloaded and hit Continue. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click Import.

    4. +
  4. +

    After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit Start. Once started up, you will see DrRacket and Firefox on the VM’s desktop.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/06/05/report-pliss-2017/index.html b/blog/2017/06/05/report-pliss-2017/index.html new file mode 100644 index 00000000..630360b8 --- /dev/null +++ b/blog/2017/06/05/report-pliss-2017/index.html @@ -0,0 +1,223 @@ + + + + + + Report: PLISS 2017 + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Report: PLISS 2017

+

+ :: pliss, event

+

By: Ming-Ho Yee

+
+ +

Two weeks ago, I attended the first Programming Language Implementation Summer School, held in beautiful Bertinoro, Italy.

+ +

The goal of PLISS was “to prepare early graduate students and advanced undergraduates for research in the field,” and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.

+ + +

The Talks

+ +

With the amount of technical content at PLISS, there was easily something for everyone. Jan Vitek and Laurence Tratt gave lectures that included hands-on exercises where we worked on JITs. Suresh Jagannathan dived into the operational semantics of a distributed system, so we could reason about different weak consistency models. Francesco Logozzo gave us a whirlwind tour of abstract interpretation.

+ +

Most of my favorite talks included some form of extra content, such as exercises, live-coding presentations, or demos. I found it really helpful to write actual code and apply what I had just learned, or to look at some concrete examples. The examples and exercises also helped with the pacing, as actively listening to four 90-minute talks every day is exhausting!

+ +

Off the top of my head, these were some of my favorite talks:

+ +
    +
  • +

    Dynamic Programming Language Implementation with LLVM, by Petr Maj, Oli Flückiger, and Jan Vitek. As the first talk of the summer school, this was a gentle introduction for the rest of the week. We had exercises (with intentional bugs to make us think!), and also brief overviews of intermediate languages, static analysis, and garbage collection. These three topics would later show up in more detail.

    • +
  • +

    Micro Virtual Machines, by Steve Blackburn. This talk covered background information on virtual machines, and also the Micro VM project that Steve’s group has been working on. A lot of the material was already familiar to me, but I still enjoyed the talk, and even got a few ideas for the project I’m working on!

    • +
  • +

    Static Analysis, by Matt Might. Matt’s talk was based on one of his articles and an older talk he’s given. Impressively, the entire example was live-coded, with only a single mistake!

    • +
  • +

    Testing Language Implementations, by Alastair Donaldson. This was an entertaining talk, since Ally showed multiple examples of crashing compilers, and causing other kinds of mischief by triggering compiler bugs.

+ +

If you’re disappointed that you couldn’t see these talks, don’t worry! The talks were recorded and will be posted very shortly.

+ +

The People

+ +

But there’s more to PLISS than the talks. I’m referring to networking, or the opportunity to get out and talk to other people about research.

+ +

As an early graduate student, I’ve been given a lot of advice about talking to people at conferences and the importance of the “hallway track.” I still have difficulty doing this at an actual conference, like PLDI or ECOOP. When there are hundreds of attendees, or when people already know each other and are in conversation groups, I find it difficult to approach them.

+ +

This was not the case at PLISS. There were fewer attendees: about fifty students and a dozen speakers. There was a good mix of undergraduate, master’s, first-year PhD, and more senior PhD students. All our breakfasts, lunches, and breaks were together, so we would see the same people again and again, and inevitably start to learn each other’s names. The speakers would also be among us, and there was a good ratio of speakers to students for discussions and mealtime mentoring.

+ +

I had many opportunities to practice my “research pitch.” I talked to senior students and got advice. I talked to junior students and gave advice. Two different people I talked to about my research pointed me to the same paper to read. I found another student who was working with IFDS, an algorithm I have spent much time trying to understand. And, one day at lunch, my table discovered that we were all working on static analysis!

+ +

As much as I enjoyed the talks, I think the best part of PLISS was meeting and talking to other people. You can replace talks with videos (but you lose the speaker-audience interaction), and you can replace conversations with other forms of communication. But there isn’t really anything that can replace the serendipity of bumping into someone with a shared interest.

+ +

The Location

+ +

Actually, the other best part of PLISS was the location. Italy is a beautiful country with delicious food. And Bertinoro is a small town on the top of a hill, with a breathtaking view of the surrounding area. The lectures were held in a castle at the top of the hill (photo credit: Steve Blackburn). The speakers lived in the castle for the week, while the students lived in the former monastery (seems fitting), which has been renovated into a university residence.

+ +

Here are my two favorite pictures I took (click for full size):

+ +

View from the castle

+ +

Panorama

+ +

Steve Blackburn has more pictures posted on the PLISS website.

+ +

Final Thoughts

+ +

PLISS was a wonderful event. Many thanks need to be given to the speakers, organizers, and sponsors, for making this possible!

+ +

If and when there is a second PLISS, I highly encourage students to apply! You will learn a lot from the lectures, from talking to the speakers, and meeting other students. And if it’s in Bertinoro again, you can enjoy the weather and nice view!

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/06/05/syntactic-parametricity-strikes-again/index.html b/blog/2017/06/05/syntactic-parametricity-strikes-again/index.html new file mode 100644 index 00000000..4eb00a18 --- /dev/null +++ b/blog/2017/06/05/syntactic-parametricity-strikes-again/index.html @@ -0,0 +1,440 @@ + + + + + + Syntactic parametricity strikes again + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Syntactic parametricity strikes again

+

+ ::

+

By: Gabriel Scherer, Li-Yao Xia

+
+ +

In this blog post, reporting on a collaboration with Li-Yao Xia, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely “syntactic” way, by looking at the structure of canonical derivations. More precisely, I prove that \( +\newcommand{\List}[1]{\mathsf{List}~#1} +\newcommand{\Fin}[1]{\mathsf{Fin}~#1} +\newcommand{\Nat}[1]{\mathbb{N}} +\newcommand{\rule}[2]{\frac{\displaystyle \array{#1}}{\displaystyle #2}} +\newcommand{\judge}[2]{{#1} \vdash {#2}} +\newcommand{\emptyrule}[1]{\begin{array}{c}\\[-1em] #1 \end{array}} + ∀α. \List α → \List \alpha +\) is isomorphic to \( + Π(n:\Nat{}). \List{(\Fin{n})} +\) where \(\Fin{n}\) is the type of integers smaller than \(n\), corresponding to the set \(\{0, 1, \dots, n-1\}\).

+ + +

Context: Last week I had the pleasure of visiting UPenn, where I had many interesting discussions with various people. It was also an occasion to temporarily resume a discussion/collaboration I have with Li-Yao Xia, who is currently an intern there, and Jean-Philippe Bernardy, about testing polymorphic programs and its relation to canonical representations for System F.

+ +

During one of our blackboard discussion, Li-Yao and I did a manual proof of a cool result: we proved a parametricity theorem for \(∀α. \List α → \List α\) using syntactic methods, namely proof search among canonical proofs. (This is an idea that I have been playing with since the last year of my PhD thesis, where I unsuccessfully tried to extend my work on canonical forms for the simply-typed lambda-calculus to polymorphism. It is here worked out on an specific example, but my end goal is to turn the manual reasoning into an algorithm.)

+ +

You may wonder, first, why the isomorphism holds. The idea is that a polymorphic function of type \(\List α → \List α\) cannot inspect the elements of the input list; it can only use them in the resulting list, possibly duplicating, reordering or dropping some elements. On any input list of size \(n\), the behavior of the function can be described by a list of indices in \([0; n-1]\). For example, if the input \([x, y, z]\) (for some values of \(x, y, z\)) gives the output \([y, y, x]\), then this relation will hold on any value of \(x, y, z\), as the function cannot inspect their value or even test them for equality. The behavior of this function on lists of size 3 can be fully described by the list of indices \([1, 1, 0]\). Its whole behavior is then uniquely determined by one such list for each possible size:

+ +

\[ + ∀α. \List α → \List α \quad≃\quad Π(n:\Nat{}). \List{(\Fin n)} +\]

+ +

The idea behind the “syntactic” (proof-theoretic?) proof method is the following: the set of closed values at a type \(A\) is isomorphic to the search space for canonical/normal derivations of \(\judge{}{A}\). We have tools (in particular the notion of invertible inference rules) to reason on those – in this post I will only present the reasoning informally, but it can easily be made formally precise.

+ +

We start by looking at the shape of the search space for

+ +

\[ + \judge{}{∀α. \List α → \List α} +\] or, said otherwise, of the judgment \[ + \judge{}{\List α → \List α} +\]

+ +

with a fresh/abstract type variable \(α\). (I will not be keeping opened type variables in context to avoid confusing them with hypotheses.)

+ +

Any derivation of a function type, without loss of generality (w.l.o.g), is equivalent to a derivation starting with a function introduction. This is the η-expansion rule for functions: any proof term \(e\) is equivalent to \(λx.~(e~x)\), a proof that starts with a \(λ\). So any proof can be taken to start as follows: \[ +\rule{ +\judge{\List \alpha}{\List \alpha} +}{ +\judge{}{\List \alpha \to \List \alpha} +} +\] we can, w.l.o.g, unfold the recursive type in the context (\(\List α = 1 + (α × \List α)\)): \[ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}} +\]

+ +

A derivation with a sum type as hypothesis can, w.l.o.g, be assumed to start by splitting on this pair (this is the η-expansion rule for sums): \[ +\rule{ +\judge{1}{\List α} +\quad +\judge{α × \List α}{\List α} +}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]

+ +

In the right subgoal, we can always, w.l.o.g, split a hypothesis of product type: \[ +\rule{ +\emptyrule{\judge{1}{\List α}} +\quad +\rule{ +\judge{α, \List α}{\List α} +}{ +\judge{α × \List α}{\List α} +}}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]

+ +

Now, an interesting pattern emerges. In the process of trying to prove \(\judge{\List α}{\List α}\), we have to prove the (right) subgoal \(\judge{α,\List α}{α}\). We can generalize this derivation by assuming that we start with some number \(n\) of variables of type \(α\) in the context (we write \(α^n\) for this): \[ +\rule{ +\rule{ +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n, 1}{\List \alpha} +} +\quad +\rule{ +\judge{\alpha^{n+1}, \List \alpha}{\List \alpha} +}{ +\judge{\alpha^n, \alpha \times \List \alpha}{\List \alpha} +}}{ +\rule{ +\judge{\alpha^n, 1 + (\alpha \times \List \alpha)}{\List \alpha} +}{ +\judge{\alpha^n, \List \alpha}{\List \alpha} +}} +\]

+ +

\[ +\newcommand{\llbracket}{〚} +\newcommand{\rrbracket}{〛} +\newcommand{\sem}[1]{\llbracket{} #1 \rrbracket{}} +\]

+ +

Let us write \(\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}}\) for the search space corresponding to all possible derivations of the judgment \(\judge{\alpha^n, \List \alpha}{\List \alpha}\). All the proof steps above have been done “without loss of generality” (in terms of focusing, we only used invertible rules), so they appear in any such derivation. Similarly, let us write \(\sem{\judge{\alpha^n}{\List \alpha}}\) for the space of all possible derivations of \(\judge{\alpha^n}{\List \alpha}\), then above we have proven that \[ +\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} +\quad=\quad +\sem{\judge{\alpha^n}{\List \alpha}} +\times +\sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} +\]

+ +

This equality can be unfolded at will \[ +\begin{align} +& \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} \\ += & \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} \\ += & \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}, \List \alpha}{\List \alpha}} \\ += & \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+3}, \List \alpha}{\List \alpha}} \\ += & \dots \\ +\end{align} +\]

+ +

or written as an infinite product \[ + \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} + \quad=\quad + \prod_{k \in \Nat{}}{\sem{\judge{\alpha^{n+k}}{\List \alpha}}} +\] and, in particular, \[ +\begin{align} +& \sem{\judge{}{\List \alpha \to \List \alpha}} \\ += & \sem{\judge{\alpha^0, \List \alpha}{\List \alpha}} \\ += & \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ +\end{align} +\]

+ +

Now let’s look at the structure of the derivations of \(\judge{\alpha^n}{\List \alpha}\). A proof of this judgment cannot start with a “left rule”, inspecting the value of one of the \(n\) variables of type \(α\), given that the structure of \(α\) is unknown/abstract. It must start by choosing to either build the empty list or a cons cell. We write this as follows (after unfolding the type):

+ +

\[ +\rule{ +\rule{ +\judge{\alpha^n}{1} +\quad\oplus\quad +\judge{\alpha^n}{\alpha \times \List \alpha} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]

+ +

The \(\oplus\) notation between two judgments is non-standard; it means that they are not two requirements of the same proof, but two alternatives for possible proofs. All valid proofs fit that structure, and they either have a \(\judge{\alpha^n}{1}\) premise or a \(\judge{\alpha^n}{\alpha \times \List \alpha}\) premise. With this syntax, we are describing a set of possible derivations, rather than a single (partial) derivation.

+ +

Proofs of \(\judge{\Gamma}{1}\) are trivial, and a proof of a product is always, w.l.o.g, a product of proofs (in intuitionistic logic / the λ-calculus they reuse the same context), so we can decompose further: \[ +\rule{ +\rule{ +\rule{ +}{ +\judge{\alpha^n}{1} +} +\quad\oplus\quad +\rule +{ +\judge{\alpha^n}{\alpha} +\quad +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n}{\alpha \times \List \alpha} +} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]

+ +

There is exactly one possible proof of \(\judge{\alpha^n}{1}\), so its search space is \(1\), the unit set (with a single element). There are exactly \(n\) possible proofs of \(\judge{\alpha^n}{\alpha}\), so the search space is just \(n\), seen as a set, or, in type-theoretic notation, \(\Fin{n}\). We thus have the recursive equation: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +1 + (\Fin n \times \sem{\judge{\alpha^n}{\List \alpha}}) +\]

+ +

This type is either \(1\), or a \(\Fin{n}\) and itself, recursively. This is exactly a list: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +\List{(\Fin{n})} +\]

+ +

so, plugging everything together: \[ +\begin{align} +& \sem{\forall \alpha. \List \alpha \to \List \alpha} \\ += & \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ += & \prod_{n \in \Nat{}}{\List{(\Fin{n})}} \\ +\end{align} +\]

+ +

Post Scriptum

+ +

Some of reasoning steps above can be formulated in a way that is less clear but more familiar, as a sequence of type isomorphisms. For example, the first part on \(\sem{\judge{\alpha^n, \List +\alpha}{\List \alpha}}\) can written as:

+ +

\[ +\begin{align} +& +∀α. αⁿ × \List α → \List α +\\ & += \text{(unfold List)} +\\ & + ∀α. αⁿ × (1 + α × \List α) → \List α +\\ & + = \text{(distribute × over +)} +\\ & + ∀α. ((αⁿ × 1) + (αⁿ⁺¹ × \List α)) → \List α +\\ & + = \text{(A × 1 ≃ A)} +\\ & + ∀α. (αⁿ + (αⁿ⁺¹ × \List α)) → \List α +\\ & + = \text{(A+B) → C ≃ (A→C)×(B→C)} +\\ & + ∀α. (αⁿ → \List α) × (αⁿ⁺¹ × \List α → \List α) +\\ & + = \text{(distribute ∀α below product)} +\\ & + (∀α. αⁿ → \List α) × (∀α. αⁿ⁺¹ × \List α → \List α) +\\ +\end{align} +\]

+ +

Reading this equational sequence, it may look like we had to make the right choice at each step; but the proof-search perspective reveals that there were in fact no choices, as each time we apply invertible rules (“w.l.o.g. rules”).

+ +

Furthermore, some parts cannot be derived in this style; in the latter part of the proof, the isomorphism between \(∀\alpha. \alpha^n → \alpha\) and \(\Fin{n}\), which is immediate from a proof search perspective, cannot be justified in this way. (In particular, \(A×B → C\) is not isomorphic to \((A→C)+(B→C)\).)

+ +

Going further

+ +
    +
  • +

    It is an unfortunately-little-known obvious fact that many things we associate to “free theorems” can be recovered by proof search. For example, it is much simpler to prove that the only inhabitant of \(\forall \alpha. \alpha \to \alpha\) is the identity using proof search than parametricity. I briefly discussed the idea in the section 1.3 of my 2015 article, Which simple types have a unique inhabitant?.

    • +
  • +

    If you are unfamiliar with proof search (or the LF community) and curious about what I mean by “canonical forms” and why I think this is an important idea, see my non-technical 2017 article Search for Program Structure. The problem of extending the notion of canonical forms to arbitrary polymorphic types is briefly discussed in the section 14.5 of my 2016 phd manuscript.

    • +
  • +

    If you haven’t heard of it yet, you would probably be interested in the 2010 article Testing Polymorphic Properties by Jean-Philippe Bernardy, Patrik Jansson and Koen Claessen. Li-Yao has a 2016 implementation called Metamorph that got us talking together. The problems of understanding canonical forms and testing are quite related, but yet not exactly the same…

+ +

You might also like

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/06/09/-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html/index.html b/blog/2017/06/09/-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html/index.html new file mode 100644 index 00000000..1168a047 --- /dev/null +++ b/blog/2017/06/09/-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html/index.html @@ -0,0 +1,169 @@ + + + + + + [Bridging the System Configuration Gap (Cross-Post)](https://aaronweiss.us/posts/2017-06-05-bridging-the-system-configuration-gap.html) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Bridging the System Configuration Gap (Cross-Post)

+

+ ::

+

By: Aaron Weiss

+
+ +

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/06/16/spring-2017-pl-junior-retrospective/index.html b/blog/2017/06/16/spring-2017-pl-junior-retrospective/index.html new file mode 100644 index 00000000..c3055a6d --- /dev/null +++ b/blog/2017/06/16/spring-2017-pl-junior-retrospective/index.html @@ -0,0 +1,229 @@ + + + + + + Spring 2017 PL Junior Retrospective + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ + +

Format

+ +

As discussed in last fall’s retrospective, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.

+ +

One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.

+ +

In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!

+ +

Synthesis

+ +

The first paper on program synthesis that we looked at was A Deductive Approach to Program Synthesis by Manna and Waldinger. We chose this paper because it’s old and has a lot of citations so it’s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.

+ +

The next paper was Programmatic and Direct Manipulation, Together by Chugh, Hempel, Spradlin, and Alders, which presents the Sketch-n-Sketch system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.

+ +

We concluded our program synthesis segment with Type-and-Example-Directed Program Synthesis by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.

+ +

While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior

+ +

Sequent Calculus

+ +

We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by Herbelin. Gabriel’s thesis (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.

+ +

It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.

+ +

An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.

+ +

Logic Programming

+ +

The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what cut does).

+ +

The second session presented the paper The Semantics of Predicate Logic as a Programming Language. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.

+ +

In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming

+ +

Bonus Rounds

+ +

We also used a few weeks to read and discuss specific papers as a group.

+ +

The first paper we read was Cardelli’s Typeful Programming. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.

+ +

It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.

+ +

Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.

+ +

After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli’s language seem rather mundane. However, it’s likely that at the time he published it, these ideas were significantly newer and much less widespread.

+ +

The other paper we read as a group was Alan Kay’s The Early History of Smalltalk. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.

+ +

Final Thoughts

+ +

Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!

+ +

Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.

+ +

Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/06/24/quotes-and-stories-from-turing-50/index.html b/blog/2017/06/24/quotes-and-stories-from-turing-50/index.html new file mode 100644 index 00000000..9d92e305 --- /dev/null +++ b/blog/2017/06/24/quotes-and-stories-from-turing-50/index.html @@ -0,0 +1,1042 @@ + + + + + + Quotes and Stories from "Turing 50" + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Quotes and Stories from “Turing 50”

+

+ :: dear diary

+

By: Ben Greenman

+
+ +

The ACM recently hosted a celebration of 50 years of the A.M. Turing award. These are some notes and thoughts from the event, including how Fred Brooks once rented a bus, Don Knuth’s outrageous implementation of batch processing, and Judea Pearl’s theory of homo sapiens.

+ + + + +

Conventions / Disclaimers:

+ +
    +
  • +

    The blockquotes below are paraphrased, may be incorrect, and may be incorrectly attributed. Make sure to watch the ACM’s live stream before quoting anything here!!!

    • +
  • +

    Section-breaks are labeled as “panel”, “talk”, “question”, etc.

    • +
  • +

    This is intentionally “bad writing” in the Peter Lee sense (see below) — primarily “what I saw”, very little about “what I thought and felt”. A summary in my own words just wouldn’t do justice to the panelists.

    • +
  • +

    The “Augmented Reality” session was my favorite.

+ +

Opening Remarks

+ +

Alan Turing is with us today

+ +

At the start of the event, the emcee unveiled a bronze bust of Alan Turing. This statue was on display at center stage during the whole event.

+ +

It’s a good sculpture and it’s good we remember Alan Turing, but I’m sad that the ACM would encourage this kind of idol-worship. Let’s not forget Turing’s excellent teachers and colleagues!

+ +

talk: Impact of Turing Recipients’ Work

+ +
Barbara Liskov
+ +
+

the first awards recognized achievements in the standard fields of theory, AI, and systems

+

hostile environment around the first awards, trepidation about future awards

+

with Unix, Ritchie and Thompson got the design right

+

Niklaus Wirth: “If I understood how important Pcode was, I would have spent more time designing it”

+ +

thoughts

+ +

What is “systems” — does that even have a definition? And Unix is definitely NOT an example of a “right design”; rather it’s a landmark of worse is better design.

+ +

panel: Advances in Deep Neural Networks

+ +

Stuart Russell

+ +
+

I work in all areas of AI except for deep learning

+ +

Judea Pearl

+ +
+

I am a foreigner in this field … left because human beings are not good at handling information … people are very good with causal inference, not with statistical inference … deep learning is statistical

+

there is a very old existence proof, homo sapiens took over the planet … I believe because they had an internal model of their environment … a drawing of a lion with wings is evidence of this model, you have to have such a model before you can experiment with it and imagine … snakes have superb optics, result of a long evolution process … very specific but they cannot build eyeglasses … humans have an internal model, can build a market based on promises and build large communities based on promises

+

I see four levels … second level is predicting events, if I do X then what? … third level is counterfactual, if I did things differently then how would the outcome change … very hard to advance between levels, are we working to help machine learning ‘level up’?

+

data science is about the relation between data and reality … data alone is not data science

+ +
Michael Jordan
+ +
+

today we can’t think without holding a piece of metal

+

machine learning is part of computer science rather than AI … AI is about how to make human … machine learning is about allocating resources … matrices are not all of human intelligence … neural nets are part of a wider toolbox … too much hype in NLP its just syntax

+

huge gap between syntax and semantics … chat bots are just syntax, don’t learn … faking intelligence with neural nets, so well that you can build a company …

+

real metric is task completion

+

if I say ‘a GLEEB walked across the airport’ then true intelligence can make a lot of educated guesses about a ‘GLEEB’ without any other context

+ +
Fei-Fei Li
+ +
+

I disagree, ML is part of AI … understanding intelligence and making intelligent methods for solving AI problems

+

to quote Churchhill ‘its not beginning of end, not end, not beginning of end, probably end of beginning’

+

todays AI powered by hardware and data

+

AI cannot yet find our keys

+

quote: ‘todays AI is making a perfect chess move while the world is on fire’ … ignores context

+ +
Stuart Russell
+ +
+

Turing … a program is a mathematical object … math community did not recognize this

+

lots of grad student descent … tuning to get performance … deep learning is neglecting the problem of exponential data … deep learning is just circuits, circuits lack expressive power … a human can process data from CERN but a neural net cannot, need to know physics

+

probabilistic programming, somewhat under the radar, maybe on the right track … 10-line program running/generating/updating a large network of possibilities … more composable and flexible

+ +
Ilya Sutskever
+ +
+

why I like deep learning … philosophically satisfying … the hypothesis class is a circuit … powerful hypothesis class not too many parameters … can actually find circuits … ‘violates all theory’ … really amazing … humans can see and hear pretty fast, even though our neurons are pretty slow, perhaps because we do a massively parallel process that doesn’t take many steps … works well enough to be useful

+

models e.g. for vision are very hard to understand … fight fire with fire … incomprehensible solution to incomprehensible problem

+ +
Raquel Urtasun
+ +
+

the breakthrough in neural nets is not algorithms … it is tricks, hardware, and grad students

+

with neural nets we forget about modeling, uncertainty, and prior knowledge … perception is a canonical example

+ +

question: boundaries

+ +
Judea Pearl:
+ +
+

glad to see people in deep learning understand its limitations … is there a clearer definition of the boundaries? Are you worried about bridging the levels factual/inferential/counterfactural?

+ +
Michael Jordan
+ +
+

the big problem is decision making under uncertainty

+ +
Fei-Fei Li
+ +
+

cognition is a hard problem

+ +
Judea Pearl
+ +
+

do you have a clear idea of the boundaries?

+ +
Michael Jordan
+ +
+

neural nets use back-propagation … its non-modular, sad fact … performance and explainability is the tradeoff … then again people are non-modular

+ +
Stuart Russell
+ +
+

AlphaGo is not deep learning … basically an improved version of the machines Arthur Samuel made in the late 1950s … the interesting code is in C++ … rules of go, next moves, searching future states … depends on transitive closure

+ +
Judea Pearl
+ +
+

can AlphaGo take advice from a human?

+ +
Michael Jordan
+ +
+

not currently, but that would be a new policy to add to the toolbox … just as neural nets are one tool within AlphaGo

+ +
Raquel Urtasun
+ +
+

no reason to ask if deep learning is going to solve all problems

+ +

question: education?

+ +
Judea Pearl
+ +
+

indeed, what DO you teach in your neural networks classes?

+ +
Fei-Fei Li
+ +
+

… chain rule, Taylor expansion

+ +
Judea Pearl
+ +
+

teaching is communicating truths … what is true about neural nets? what are some things that will definitely not happen?

+ +
Stuart Russell
+ +
+

Peter Norvig and I have a problem with our AI book … chapter on vision, chapter on speech, will probably post just point to the neural nets chapter … we don’t really understand! … really selling students short

+ +
Fei-Fei Li
+ +
+

in labs we talk about what we cannot do … we all have open problems

+

Stuart I hope you have a very good author for the chapters. There are so many open problems to communicate to students!

+ +
Michael Jordan
+ +
+

CS cirriculum needs more statistics, inferential thinking … revise the whole cirriculum bottom-up to weave this in

+ +

question: could a neural net fix my phone without breaking it?

+ +
Judea Pearl
+ +
+

right! big problem that neural nets have no internal model to manipulate

+ +

question: generalizability?

+ +
Ilya Sutskever
+ +
+

special-purpose vs. general purpose solution depends on the problem … most things we give special-purpose solutions … I guess if you wanted to automate a mathematician that would need to be general

+ +
Stuart Russell
+ +
+

always argue with your self … try to break what you’ve built … there’s a system that plays video games just using the pixels on screen as hints … it’s very good at mazes; if a newborn baby learned to play maze games in 2 hours that would be amazing! … does the system scale? absolutely not

+ +

thoughts

+ +

When Michael Jordan said “people are non-modular”, I think he means that people are able to break abstraction barriers when needed.

+ +

panel: Restoring Personal Privacy without Compromising National Security

+ +
Joan Feigenbaum
+ +
+

… wikileaks … russian hackers … social emergency …

+ +
Whitfield Diffie
+ +
+

everything I say today is copyleft

+

its a misunderstanding to talk about a conflict between security and privacy … two aspects … problem goes back to feudalism … the right to build a castle was granted by the king … on one hand a castle improves national security … on the other hand a castle can be used to attack the king … technology is upsetting the basic notion of private vs. public security … governments cannot protect citizens and cannot protect themselves … extremely difficult to prove that a small process is secure

+

exceptional access makes it more complex

+ +
Paul Syverson
+ +
+

major concern are national security threats and ability of authorities to confound threats … analogy to printing press … proclimation of 1635 that only state messengers can carry letters … 1663 treatise by the national censor, no printing house can have a back door … the general topic is very old … title of this session isn’t very good, the real dilemma is investigation vs privacy

+ +
Bryan Ford
+ +
+

code is law for better or worse, tech is not a tool like a watch … tech can monitor us and decide when it works … tech is government, not obedient tools … the mind is a warrant-proof space … 5th amendment rights should extend to wearables

+ +
Nadia Heninger
+ +
+

cannot divorce the security/privacy issues from the current political context … the serious vulnerabilities are not in math … they are in users and implementors

+ +

question: back doors

+ +
Joan Feigenbaum
+ +
+

perhaps we should explain what a back door is

+ +
Bryan Ford
+ +
+

agency keeps a master key in escrow

+

non-lawyers can and should take a stand on basic issues

+

there are legitimate warrant-proof spaces … electronic extensions of the mind need to be recognized as warrant-proof spaces

+

the set of authorities with backdoor access should change as I travel between countries … but this will lead to a global race to the bottom

+ +
Whitfield Diffie
+ +
+

germany has a law against sex tourism (committed by German citizens visiting other countries) … neither government will be willing to lose backdoor access

+ +
Nadia Heninger
+ +
+

technical reasons against backdoors … (1) ‘weak crypto’ was implemented, nobody turned it off, is now breakable by anyone in 2015 … (2) Juniper used non-default crypto parameters, someone (inside?) changed the parameters … (3) attackers exploit back doors

+ +
Paul Syverson
+ +
+

quote ‘you can put a man on the moon, surely you can put a man on the sun’

+ +
Whitfield Diffie
+ +
+

trouble is getting him back safely

+ +
Bryan Ford
+ +
+

I think back doors are okay, but not for personal devices … need public lab and transparent processes, need separation of powers … prosecutors are getting cases thrown out because courts do not accept their backdoors … there is a place for transparent back door tools

+ +
Nadia Heninger
+ +
+

politicians are rarely technical people

+ +
Bryan Ford
+ +
+

tech is not a set of policy-neutral tools, need to address gap of understanding

+ +

question: ???

+ +
Whitfield Diffie
+ +
+

we don’t know how to build good crypto programs … opponents are debugging our programs with different goals … we’re trying for-all-paths safety (universal) … they’re trying exists-bad-path (existential)

+ +
Bryan Ford
+ +
+

cybersecurity market is a lemon market

+ +

question: how to advise

+ +
Joan Feigenbaum
+ +
+

question from audience ‘I am an advisor to a company working with nuclear energy, they are terrified of being attacked, how should I advise them?’

+ +
Whitfield Diffie
+ +
+

a network like that is probably separated enough to be safe … the problem is being safe AND connected to the web

+ +
Bryan Ford
+ +
+

because the internet of things

+ +

question: what should the ACM do?

+ +
Nadia Heninger
+ +
+

maybe we need increased regulation, the ACM could help bring experts together

+ +

question: what is true security

+ +
Paul Syverson
+ +
+

it’s all the same thing … gets labeled differently … just trying to control which bits can go where and who gets to read them

+ +
Nadia Heninger
+ +
+

security is the absense of being violated

+ +
Paul Syverson: no true > security, need to consider context
+ +
Joan Feigenbaum
+ +
+

problem of our community, have strict standards, may be unrealistic … maybe a lot more tolerance in practice than our model accepts

+ +
Paul Syverson
+ +
+

security and privacy are environmental problems

+ +

question: can we stop the needle-in-haystack search for vulnerabilities?

+ +
Paul Syverson
+ +
+

need to build in security from the start

+ +
Bryan Ford
+ +
+

need rule of law, transparency, separation of powers

+ +
Whitfield Diffie
+ +
+

stop delaying, instead of spending $$$ on fixing problems, we should invest in solving the fundamental issues

+ +

panel: Preserving our Past for the Future

+ +

Note: I was volunteering during this session; quotes are sparse

+ +
Mahadev Satyanarayanan
+ +
+

the running system is the total documentation … there are too many details for prose to capture

+ +
??
+ +
+

running old code has a danger of running old bugs

+ +
??:
+ +
+

what not to save? … it’s very hard to tell in advance

+ +
Mahadev Satyanarayanan:
+ +
+

there is no absolute censor in a world with caching

+ +
Brewster Kahle
+ +
+

asking UNESCO to solve the problem is unrealistic … need to empower the fanatics, given them tools to preserve data

+ +

thoughts

+ +

I totally agree with the “empower the fanatics” sentiment. Today, because of “volunteer librarians”, I think we’re doing pretty well about preserving the past. Suppose I found an old PowerPoint file. I’m sure I could find a way to read it with help from the internet — either by searching Google, pirating an old version of PowerPoint, or asking online forums. So personally I’m not worried about losing data we have currently; I’m more worried about the future, the internet becoming “less chaotic”.

+ +

The panel raised a good question about how to preserve research and encourage reproducibility. A .pdf or .tex document is not enough; a virtual machine is okay. Really I think we need a stronger cultural emphasis on literate programming and a mature library like TeX to help authors store and share their work. The Gamma seems on the right track.

+ +

I was surprised that the panel did not discuss search, version control, and the ACM’s open-access policy.

+ +

panel: Moore’s Law is Really Dead: What’s Next?

+ +

Note: I was volunteering during this session

+ +
Butler Lampson
+ +
+

there’s plenty of room at the top … with Moore’s Law we got improvements at the bottom of the software stack, everything above got to benefit and it was easy to integrate the changes … there’s lots of opportunities to trim fat in the middle/top of the software stack … these improvements will be harder to integrate, but there’s lots of opportunities

+ +
Margaret Martonosi
+ +
+

By the way, don’t believe the brochure that says I’m at Google. My affiliation is Princeton, Google and I are just friends.

+ +
Butler Lampson
+ +
+

important to distinguish approximate vs. precise software … precise software has a specification and the customer cares about that specification … approximate software doesn’t have a hard spec, just needs to approximately work … the web is approximate, it doesn’t work and it doesn’t need to! … windows is precise, definitely has a spec and users definitely care

+ +

thoughts

+ +

The recording of this panel should be good; it was very lively, very practical. And the first audience question (by David Patterson) was “an A+ question”.

+ +

The panel reminded me of a talk by Yale Patt about “the end” of the Von Neumann architecture. His opinion is future computers will be Von Neumann machines that rely on “accelerators” like a GPU — computer organization is not going to change, but will expand to have a bigger toolbox. So sure, Moore’s Law is dead, but there are many opportunities to make computers faster at places other than the bottom of the software stack.

+ +

panel: Challenges in Ethics and Computing

+ +

Note: I was volunteering during this session

+ +
Raj Reddy
+ +
+

there are more slaves in the world currently than there were in the US during the civil war … here is one way technology could help, by giving everone a device to record their location … if someone’s time and location is constant, they may be held against their will

+ +
??
+ +
+

do you believe every problem has a technological solution?

+ +
Noel Sharkey
+ +
+

yes the training set may be biased against people similar to me, but I want you to consider my case as an individual

+ +
??
+ +
+

a very nice Washington Post article

+ +
??
+ +
+

whether to encrypt the back hall

+ +
Raj Reddy
+ +
+

we can sit here and wring our hands, but nothing will come of it unless it is written in the US constitution

+ +

thoughts

+ +

I did not enjoy this panel. This is an ACM event, not a United Nations event. An ACM-sponsored panel about social and political problems should look for constructive ways that computer science can address these problems. Raj Reddy tried to give constructive solutions, but the panel seemed more interested in complaining about how hopeless things are.

+ +

The comment by Noel Sharkey about “consider me as an individual” was something I hadn’t thought about. Instead of worrying about biased datasets, let’s use technology to collect data on an individual instead of abstracting a person by their race, age, or neighborhood.

+ +

talk: Computer Science as a Major Body of Accumulated Knowledge

+ +
Donald Knuth
+ +
+

don’t applaud me, just read my books

+

at the time, computer science was AI, numerical analysis, and programming languages

+

a colleague said ‘I will believe that computer science is a science when it has 1000 deep theorems’ … I am not sure what a deep theorem is but I think its different from what’s proven by deep learning

+

great privilege that we can invent the problems we work on … imagination … physicists can only guess the size of the sun

+

I’ve always believed computer science and math are two parallel subjects … sometimes you hear people wondering if one subsumes the other

+

when I won the Turing Award, the prize money was about $1,000,000 less than it is today … I did get a nice Tiffany bowl that my wife and I use to serve strawberries … strawberries actually taste better …

+

very fortunate in this field … I’m completely worthless as an economic advisor … it’s a game I’ve been able to take advantage of

+ +

question: how could you offer to pay for TeX bug reports?

+ +
Donald Knuth
+ +
+

well there were many, many bugs … I stopped doubling at 32768 … brought people out of nowhere … next time I check the bug reports will be 2021 … someone is queueing the bugs reports … I believe strongly in batch rather than swap-in/swap-out … last time I checked reports was 2014 so 2021 will be next

+

TeX was a literate program, and it helped that I wrote ‘The Errors of TeX’ about the first N bugs

+ +

question: do you think computers will become good composers of music? do you see a role for computer-assisted proving?

+ +
Donald Knuth
+ +
+

Yes in part, assisted is the key word … I have a program running now that I hope will help me prove a theorem

+ +

question: favorite algorithm?

+ +
Donald Knuth
+ +
+

Tarjan’s strong components … short deep useful

+ +

question: thoughts on AI, computers taking over?

+ +
Donald Knuth
+ +
+

I get scared when I see Stuart Russell making assumptions based on people acting rationally … then you look at election results

+ +

question: if you could start over and do things differently, what would you change?

+ +
Donald Knuth
+ +
+

I would use decimal internally in TeX instead of binary

+ +

question: how to record history?

+ +
Donald Knuth
+ +
+

a video ‘Lets not dumb down the history of CS’ … used to be history of algorithms … trouble with funding … the history is nothing that a non-CS person could not understand … the whole field of history changed from internal to external … historians need to be external to get published in journals … no CS department supports a historian … recently read a dissertation about the ALGOL 60 copmiler … very careful, describes data structures and organization … this kind of thing is what deserves to be called history

+ +

question: teachers

+ +
Donald Knuth
+ +
+

hardest thing for me is choosing between two hypotheses (1) could teach this to anyone (2) only 2% of the world is geeks … suppose the second is true then you can’t talk about how to teach if the teacher is not in the 2% …

+

the newest issue of CACM has a fun typo, ‘deep earning’

+ +

panel: Quantum Computing: Far Away? Around the Corner? Or Maybe Both at the Same Time?

+ +
John Martinis
+ +
+

goal to have a 45–50 qbit machine … 1 error per 1000 operations … to test, run sample algorithm, chart output vs. a classical supercomputer … got to be a supercomputer to finish the computation in time

+ +
Andrew Yao
+ +
+

I’m a believer … one suggested benchmark is to factor 1000-digit numbers … impossible to attain … need to expore new possibilities, take physics attitute

+

CS did well attracting attention to quantum … science should be more open … share results between physics chemistry CS … don’t just stick to your specialized conferences

+

CS departments reception to quantum is less than satisfactory … 15 years ago, maybe 4 or 5 universities … now, maybe 7 or 8 .. China doing much better in this regard

+ +
Jay Gambetta
+ +
+

not useful to make analogy to anything classical … universal fault tolerance? or computation in the presence of error … either would be excellent, still a long way off

+

IBM put quantum on the cloud … picked an instruction set that tries to abstract away … have been 19 published papers on the behavior of this quantum hardware

+ +
Dorit Aharonov
+ +
+

two paths … finding algorithms, besides Shor’s algorithm … make quantum computer to realize the algorithms … finding algorithms is very difficult … information-processing point-of-view

+

error correction still small scale … can we use entanglement between probes to improve accuracy?

+ +
Umesh Vazirani
+ +
+

_different goals … maybe you want perfect Qbits for a perfect Hilbert space … reality is a noisy space … short run, how to compute with noise … how to correct errors …

+ +
Jay Gambetta:
+ +
+

_those 2 paths are the same to me … we want larger devices with fidelity

+

lets build hardware see where goes … exciting prospect, computer scientists will explore what they can do with these erroneous qbits … that’s why IBM has the instruction set open to the community

+ +

question: why isn’t adding 10 qbits only 10x harder?

+ +
John Martinis:
+ +
+

building infrastructure to scale … not just grad student code … we’re all good coders using standard industry practices for coding

+ +
Jay Gambetta
+ +
+

fidelity is hard to achieve

+ +

question: both IBM and Google use superconducting storage?

+ +
John Martinis
+ +
+

superconducting scales … ion traps harder to scale, but we still watch, keep eye on data

+ +

question: education

+ +
John Martinis
+ +
+

I like talking to engineering colleges … physics and engineering need to work together

+ +

question: is quantum going to change programing languages?

+ +
Jay Gambetta
+ +
+

yes very different to handle errors … current challenge is building an abstraction over the superconducting hardware

+ +
John Martinis
+ +
+

hoping to first expose hardware, then get a model, eventually a language

+ +
Dorit Aharonov
+ +
+

need to start with more algorithms

+ +

question: what would Feynman do?

+ +
John Martinis
+ +
+

experiments!

+ +
Dorit Aharonov
+ +
+

yes he’d tell us to keep playing, and play with us

+ +

panel: Augmented Reality: From Gaming to Cognitive Aids and Beyond

+ +

Peter Lee starts off wearing a headset.

+ +
Ivan Sutherland:
+ +
+

I can tell you how VR started. Bell Helicopter company wanted to land at night … put an infrared camera on the landing site and a display in the cockpit … to test they used the roof of their building … one day an observer in a Bell office is watching, though the camera, two men playing catch on the roof … one player threw the ball at the camera and the observer ducked … he had identified his position with the camera … my observation was that you didn’t need a camera, could substitute a computer … the rest is history

+ +
Yvonne Rogers
+ +
+

my goal is to augment people … Englebart very inspiring … ok 2 stories … (1) a student of mine wanted to help picky eaters … computer vision for when they tried to hide peas under the plate … projected colors onto the peas, called them ‘disco peas’, kids were distracted enough to get over their aversion … children and parents got involved, new social interaction … (2) opera makeup for schoolchildren, virtually getting into character … teenage boys in the classes got to try makeup for the first time … singers found it useful for rehearsals

+ +
Peter Lee
+ +
+

I feel socially awkward wearing this headset, but I have some of my slides here … making a wearable headset requires huge effort … research prototypes can be uncomfortable … a product needs to be perfect and its very hard to do perfect … one goal, give Lowe’s VR to demo a virtual kitchen … Case Western anatomy class used virtual cadaver, great collective experience

+ +
Fred Brooks
+ +
+

two stories … (1) Henry Fuchs 1998, working with breast surgeon, try augmented reality to improve the precision of biopsy probe insertion … 2 years to a working prototype, hard to track surgeon’s eyes, display where probe is, where ultrasound is, provide low latency … one day trying on live patient, worked 100% perfect probe right on the mark, jubilation … then the doctor had to tell the patient ‘yes it is really cancer’ … (2) a challenge, augmented reality EMT training … real teams, virtual patient, virtual surround … track real tools, 8 eyes, 8 images, team needs to interact

+ +

question: what are current uses of augmented reality?

+ +
Ivan Sutherland
+ +
+

the pilot of a jumbo jet typically has 1 hour flight experience before he flies for the first time, but extensive training in a flight simulator

+ +
Fred Brooks
+ +
+

the best AR

+ +
Ivan Sutherland
+ +
+

once I was in a flight simulator with the chief pilot … and he turned to me and asked ‘have you ever experienced a slow roll in a 747?’ … a slow roll is a twisting motion, a very benign maneuver, constant one-G pressure the plane doesn’t know its upside down … ‘here we go’ and suddenly the world inverted … I remarked that it was certainly impressive, but didn’t you treat the simulator as a real experience, and never attempted anything you would not do in reality? … ‘that is true, but I am the chief pilot’

+ +
Peter Lee
+ +
+

construction, architecture

+ +
Fred Brooks
+ +
+

where’s the ‘augmented’?

+ +
Peter Lee
+ +
+

whether augmented or virtual

+ +
Fred Brooks
+ +
+

yes we did my kitchen that way, made my wife sick when she tried it

+ +
Peter Lee
+ +
+

surgical

+ +
Fred Brooks
+ +
+

still sounds virtual

+ +
Yvonne Rogers
+ +
+

displays on a car, superimposed directions on the tarmac … one of the users took a vacation and had to use the old GPS technology … found it very difficult to go back

+ +

question: AR tools for developers?

+ +
Blair MacIntyre
+ +
+

can developers write apps for the Microsoft Hololens?

+ +
Peter Lee
+ +
+

we belive in experience, anything we can do to foster experiences is good

+ +
Fred Brooks
+ +
+

faking things … subtle and important … I remember using a flight simulator, navigating the runway, and I turned my head to see if my wing was going to clip a plane … turned and there was nothing there … emotional shock to leave the simulation, I had been flying for 1 hour

+ +
Ivan Sutherland
+ +
+

pilot training is an early adopter because the cost of real planes is so high, impossible to train for emergency situations

+

the ultimate goal, you can sit in a virtual chair … and if the chair has handcuffs you cannot get up … a virtual bullet is lethal … probably impossible because bits don’t weigh anything … you know Ben Franklin invented augmented reality, eyeglasses … the desire outweighs cost … I cannot see the audience here, maybe it would be good if I had a headset! but Peter took his off

+ +
Peter Lee
+ +
+

because of my slides I couldn’t see the audience, but then without the headset I couldn’t see them either

+ +

question: any challenges to suggest to the audience?

+ +
Peter Lee
+ +
+

if we had holographic transport, we wouldn’t need planes!

+ +
Yvonne Rogers
+ +
+

maybe, but you need to give attendees a physical presence … smell, touch

+ +
Ivan Sutherland
+ +
+

what makes us willing to work together? I had a collaboration with three people … all in different locations .. communicated with a phone … worked perfectly, because we had worked in the same location first and knew one another so well … how to get to that point, where a simulation could be a useful tool … another good observation by Fred Brooks, given a domain X ask how good does the simulation need to be for X … Licklider told me, you’d need damn good fiction to land someone on the moon, the simulation would need to provide every detail … for flight simulation the user’s imagination can fill some gaps, a pilot can recognize an aircraft carrier from a rougher picture

+ +
Fred Brooks
+ +
+

at IBM I once hired buses to bring the Poughkeepsie secretaries to the main office … the secretaries at the two offices only knew one another from the phone … this one lunch did so much good … only $75 to rent a bus

+ +
Peter Lee
+ +
+

how important is it to shake hands, to bump into a table?

+ +
Yvonne Rogers
+ +
+

for this conference, I think the live stream is getting a better experience because the cameras zoom in on us, the panelists … the audience in the back cannot see us, only a picture of us on the monitors

+ +
Peter Lee
+ +
+

_one excellent video game, starts in the dark, you hear a voice … turn around and there’s a character sitting on a chair … if you rearrange your furniture he finds a new seat …

+ +
Yvonne Rogers
+ +
+

games are a great example … Pokemon Go … Apple jusr released an app toolkit … need to get those in schools, in the hands of kids who can build with them

+ +

question: Ivan, about your ‘ultimate display’ paper, what has since surprised or frustrated you?

+ +
Ivan Sutherland
+ +
+

I wasn’t surprised because I never had any expectations … of course sticks are not real … no assumptions so no strong feelings

+ +

question: people already distracted by cell phones, how to manage all this input?

+ +
Yvonne Rogers
+ +
+

good question, how much data you can present to people … and then the problem with google glass, your companions don’t know what you are looking at … at least with snapchat glasses, you can trust the device is simpler

+ +
Peter Lee
+ +
+

good writing defines reality, bad writing reports it … with the printing press, quickly went from 30,000 books to over 13,000,000 … novels evolved shortly after, a new form of expression

+ +

question: Peter, how long do your people wear the hololens?

+ +
Peter Lee
+ +
+

hard to say … but often longer than the battery lasts

+ +
Fred Brooks
+ +
+

how long does it last?

+ +
Peter Lee
+ +
+

depends what you’re doing, 3 hours

+ +
Fred Brooks
+ +
+

that’s encouraging, we had a 30-minute cutoff because participants had enough

+ +

question: nausea

+ +
Peter Lee
+ +
+

I get nauseous in our minecraft VR … but there’s a pop-out feature where you keep playing, but the game world is in a TV set instead of around you … can pop back in when you’re feeling better

+ +
Yvonne Rogers
+ +
+

we’ve seen about 20% of the population gets nauseous

+ +
audience member
+ +
+

Dana Boyd conducted an experiment, found the nausea was worse for wemon

+ +
Yvonne Rogers
+ +
+

oculus makes me feel sick, but the hololens has never given me trouble

+ +
Peter Lee
+ +
+

have models to predict head motion, to keep the VR world steadier

+ +
Blair MacIntyre
+ +
+

I remember reading papers that measured framerate … would be interesting to revisit

+ +
Fred Brooks
+ +
+

framerate not important, its the latency that gets you … one colleague of mine, we call her ‘the canary’ because she’s so sensitive, in fact …

+ +
Peter Lee
+ +
+

talking about nausea is part of the problem, people experience it more … every time I talk about it in public my co-workers tell me to stop!

+

another cool application, there’s a hololens app to give blind people a tour of the Redmond office … you say a building and it takes you there

+ +
Fred Brooks
+ +
+

one challenge, the relative brightness of the real and virtual worlds

+ +

question: any last remarks

+ +
Ivan Sutherland
+ +
+

_I hoped from the beginning that AR would be a teaching tool … I learned that F = MA not from a book but from a large flywheel in the school’s basement … very substantial inertia … the greatest value for AR would be to show people things in a way that makes the underlying meaning clear … what color should the hydrogen atoms in a benzene ring be? the color will be fiction, but the quality of learning will depend on that fiction … challenge for content makers … what is the haptic experience of feeling bits?

+ +

Small Group Session

+ +

After the last panel, I got to attend a small group session with other students, Dick Karp, and Don Knuth. It doesn’t feel right to summarize or quote from the session, but there’s one thing I want to write about.

+ +

During the group session, I said something that I now regret. There was a brief silence as the group changed subjects, and someone suggested that we do a round of introductions. I objected, this will take so long, but in fact the introductions were a very good idea.

+ +

Normally, I don’t like introductions because they focus on names, backgrounds, and credentials. I don’t care about any of these when I’m meeting someone! Rather, I prefer to just talk and by-the-way learn about the other person(s). There’s an anaology to double-blind reviewing — the focus should be content and not credentials.

+ +

These introductions were successful for two reasons. First, they gave everyone in the room a turn to speak, and this seemed to help people join the actual discussion sooner. That was strange to me. I always feel a little nervous the first time I speak up in front of a group, but if I really feel like speaking then I can always get over this little barrier. I guess it’s not right to assume the nervousness is “little” for everyone. Second, the introductions format was “say your name and a funny fact”. This prompt by itself led to some nice conversation topics:

+ +
    +
  • Could a computer program decide whether a statement was funny or not funny?
    • +
  • What kind of humor works in a classroom? In a textbook?
    • +
  • Would this kind of introduction be acceptable in another era or culture, for instance Victorian England?
+ +

“Nice” in the sense that everyone could contribute, which was a real challenge. Even the question “does anyone have a favorite algorithm?” didn’t have much success fostering discussion.

+ +

Related: a useful greeting at the event was “what SIG are you?”. The answer was a good hint about what level of abstraction you two could best communicate at.

+ + + + + +

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/07/17/continuations/index.html b/blog/2017/07/17/continuations/index.html new file mode 100644 index 00000000..3213472e --- /dev/null +++ b/blog/2017/07/17/continuations/index.html @@ -0,0 +1,178 @@ + + + + + + Continuations + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Continuations

+

+ :: history

+

By: Ben Greenman

+
+ +

From the PRL archives:

+ +
+

It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. — Continuations by Alan Nall, Indiana University, 1983

+ + +
+ +

I first encountered this essay on continuations in a green folder in the PRL. It turns out, the author spent a semester at Indiana University working on the same fringe problem for a graduate-level programming languages course. According to the instructor: “What he said was true. He could not stop thinking about the problem the entire semester.” This essay was a kind of final exam.

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/07/19/trees-1973/index.html b/blog/2017/07/19/trees-1973/index.html new file mode 100644 index 00000000..18f6e097 --- /dev/null +++ b/blog/2017/07/19/trees-1973/index.html @@ -0,0 +1,180 @@ + + + + + + Trees, 1973 + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Trees, 1973

+

+ :: history

+

By: Ben Greenman

+
+ +

From the PRL archives:

+ +
+

I think that I shall never see a matrix lovely as a tree. — Trees, by Guy L. Steele Jr., MIT, 1973

+ + +
+ +

You might recognize the opening line from Joyce Kilmer’s 1914 poem Trees, or from Radia Perlman’s Algorhyme (published 1985).

+ +

The poem is online in at least one other place, but the copy linked above (from BYTE magazine) comes with a footnote on How this poem came to be printed.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/08/13/reviews-and-author-responses-we-should-stop-asking-for-500-word-responses/index.html b/blog/2017/08/13/reviews-and-author-responses-we-should-stop-asking-for-500-word-responses/index.html new file mode 100644 index 00000000..2458c450 --- /dev/null +++ b/blog/2017/08/13/reviews-and-author-responses-we-should-stop-asking-for-500-word-responses/index.html @@ -0,0 +1,211 @@ + + + + + + Reviews and author responses: we should stop asking for 500-word responses + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Reviews and author responses: we should stop asking for 500-word responses

+

+ ::

+

By: Gabriel Scherer

+
+ +

This year I reviewed many ICFP submissions, and got to be on the receiving end of equally many author responses (also sometimes called, somewhat combatively, rebuttals). I found that there was a large difference between the official written advice on author responses and what I, as a reviewer reading the responses, found effective. In particular, I now believe that limiting yourself to 500 words should strongly be avoided — we should even stop giving that advice.

+ + +

This year, I had the honor (and accompanying load of work) of being a Program Committee (PC) member at the ICFP conference. It was my first time being a PC member at a conference, and I found it extremely pleasant and interesting, thanks to the authors who sent us articles to review, my fellow PC members, and the ever-smiling careful balancing work of our PC chair, Mark Jones. It was also a lot of work, starting with 18 reviews to do over a month and a half, an intense PC meeting, and the new “second phase” process with the opportunity for authors and reviewers to exchange feedback on changes requested by the program committee.

+ +

There is little guidance on how to write author responses, although this blog post by Michael Hicks on pl-enthusiast is quite good. One thing that is obvious as a reviewer and is only slightly brushed in this post, however, is that author responses should not aim to fit a 500 words limit, and in fact I believe that it is a bad idea to do so.

+ +

As for most conference, the ICFP review system recommends (in writing) to authors to keep their response to 500 words (some systems also highlight words after those in red to make the point clear). Don’t do this! The least convincing responses I have seen are those that followed this recommendation.

+ +

(I have also seen at least 18*2 other reviewers read the same responses I read, most of them well over 500 words, and none of them made any comment on the length of the author responses.)

+ +

We have a frustrating situation where the explicit rule is different from the thing people do in practice. This is bad for newcomers that do not know the norms and cannot tell if ignoring the rule may hurt them. This is the point of this blog post:

+ +
    +
  • +

    If you are an author, please know that disrespecting the 500-words limit is the right thing to do. You should also know, of course, that people have limited time, so keep the main body of your response reasonably long. (I spent a day and a half on your paper already, I am willing to spend 10 additional minutes reading the response.)

    • +
  • +

    If you are a program committee chair or a Magical HotCRP Wizard, please remove this silly recommendation to keep responses to 500 words. (See an alternative proposal at the end of this post.)

+ +

My personal response format

+ +

My author responses start with general comments that elaborate on the main point I want to tell all reviewers. Then, a second part contains per-reviewer comments (one section per reviewer); it is clearly marked as skippable. Here is the skeleton of the last response I wrote:

+ +
+

We thank the reviewers for their work on our article and their detailed feedback. We start with a general discussion that responds to the salient points raised by reviewers. In a second part, we provide detailed responses to the questions/remarks of each reviewer.

+

General discussion

+

[..]

+

Specific questions/comments: review #A

+

[..]

+

Specific questions/comments: review #B

+

[..]

+

Specific questions/comments: review #C

+

[…]

+ +

For this particular response, the “General discussion” section used 1296 words according to wc -w (M-x shell-command-on-region). In the following sections, I quote the reviews to answer specific points, email-style (following the Markdown syntax that HotCRP renders properly).

+ +

Suggested wording for PC chairs

+ +

If you are a PC chair, you should remove the suggestion of respecting a 500 words limit for your conference. Here would be a suggested alternative wording:

+ +
+

Please remember, in writing your author response, that reviewers may stop reading the response at any point. We suggest having a reasonably-sized main section where you make your most important high-level comments, and clearly marked sections where you answer individual reviewer’s questions. It is not useful nor productive to answer every point of each review, you should focus on the comments that you believe the reviewers are most interested in.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/index.html b/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/index.html new file mode 100644 index 00000000..61781a89 --- /dev/null +++ b/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/index.html @@ -0,0 +1,230 @@ + + + + + + Gradual Typing Across the Spectrum, part II + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Gradual Typing Across the Spectrum, part II

+

+ :: gradual typing, PI meeting

+

By: Ben Greenman

+
+ +

Last week, Northeastern hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:

+ +

http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html

+ +

A common thread among the talks was the question: how to convert a research idea into a tool for software developers?

+ + +

In my mind, gradual typing is an answer to one instance of this question. The research idea is strong static type systems, and the software developers are the millions using dynamically typed languages. I know that static typing can make programs easier to write and maintain. The developers know that dynamic typing has benefits; moreover they know better than to migrate their code from one language to another on a whim. Gradual typing is a linguistic solution to the problem of adding the benefits of static typing to a dynamically typed language.

+ +

Enough opinions, let’s talk about the talks.

+ +

The morning session consisted of four talks:

+ +
    +
  • +

    Milod Kazerounian (UMD) spoke about upgrading the RDL type checker for Ruby with support for refinement types. The idea is to compile Ruby code and types to Rosette, and profit from SMT-assisted type checking.

    • +
  • +

    Ambrose Bonnaire-Sergeant (IU, slides) has been inferring useful Typed Clojure types through dynamic analysis of Clojure programs. His tool observes how values flow through a program at run-time, then lifts these observations into possibly-recursive, possibly-incorrect type annotations. The surprising result is that the tool quickly (1–2 seconds per unit test, I think) infers types that can help a developer start annotating a program.

    • +
  • +

    Ben Greenman (NEU, slides) explained why he is implementing a semantics for Typed Racket inspired by Michael Vitousek’s work on Reticulated Python. The “why” is “performance”. The Reticulated semantics will enforce a notion of tag soundness in kind of devils contract to improve performance.

    • +
  • +

    Preston Tunnell-Wilson (Brown, ONWARD 2017) recently sent questions about programming language design to Mechanical Turk workers. Survey says, developers have extremely diverse opinions about what they expect and what they want regarding scope, inheritance, and infix operators.

+ +

In the early afternoon, we had two talks on similar themes as the morning session:

+ +
    +
  • +

    Andre Kuhlenschmidt (IU) is exploring the design space of efficient implementations for run-time type checks. The main challenge is how to monitor higher-order data in a way that efficiently performs type checks and can help the programmer debug any failed checks. This talk presented data comparing two approaches to the program; I believe the latter, improved approach is based on coercions.

    • +
  • +

    Zeina Migeed (NEU) explained that there are many ways to adapt type soundness to a gradually typed language, and presented some data comparing Typed Racket’s generalized soudness to Reticulated Python’s tag soundness. The data suggests that tag soundness never adds an order-of-magnitude slowdown.

+ +

Next on the schedule were two talks about implementing advanced type systems in Racket’s macro expander (think: meta-level linguistic re-use, capture-avoiding substitution for free!)

+ + + +

After a short break, we heard about something completely different:

+ +
    +
  • Justin Pombrio (Brown, ICFP 2017) taught us to interpet the scoping rules of a “core” language as a preorder. Using the preorder, he then showed how to infer the scoping rules of any “surface” language based on its translation to the “core”.
+ +

Last summer and fall, Jeremy Siek hosted two REUs (research experience for undergraduates) at Indiana University. The two students gave the next talks:

+ + + +

Finally,

+ +
    +
  • Niki Vazou (UMD) presented a theory of gradual refinement types. Any “holes” in the refinements introduce a search problem; type checking attempts to solve the problem by finding a predicate that unifies a function definition and its callers.
+ +

This meeting was a great opportunity to reflect on the recent past and share opinions on what’s worth pursuing in the future. Many thanks to the participants, and to the NSF for the support!

+ +
+

If you want to know about the future, you need to ask the young people who will create it. Young people don’t know what can’t be done, and so they go ahead and do it. — Ivan Sutherland

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/08/28/closure-conversion-as-coyoneda/index.html b/blog/2017/08/28/closure-conversion-as-coyoneda/index.html new file mode 100644 index 00000000..43b90809 --- /dev/null +++ b/blog/2017/08/28/closure-conversion-as-coyoneda/index.html @@ -0,0 +1,317 @@ + + + + + + Closure Conversion as CoYoneda + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ + +

\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)

+ +

Polymorphic Closure Conversion

+ +

Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions’ free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.

+ +

For instance

+ +
let x = 3 in λ y. x + y
+ +

would be converted to something like

+ +
let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)
+ +

Can we give a type to the resulting code? The source program has type Number -> Number, but the target has a type more like

+ +
{ x: Number} × ({x : Number} × Number -> Number).
+ +

In addition to being ugly, this type is leaking irrelevant details of the function’s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just λ y. 3 + y would not even preserve typing. Not only that, but some bad code could now supply a different, well-typed value for x than allowed which could break invariants the programmer had about the function.

+ +

We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is irrelevant to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be abstract to the caller and concrete to the callee. Existential types capture this.

+ +

We can translate functions in the source of type A -> B to pairs of an environment and a code pointer, but now making the environment type existentially quantified:

+ +
∃ Γ. Γ × (Γ × A -> B).
+ +

Then the syntax of existential types ensure that all any consumer can do with the env : Γ in the pair is pass it to the code pointer with an A argument.

+ +

How do we prove that this is correct? And what does correct even mean? We’ll focus on a property called full abstraction which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:

+ +
let x = 3 in λ y. x + y
+≡
+λ y. 3 + y
+ +

To prove this we’ll show that in a language with existential types the types ∃ Γ. Γ × (Γ × A -> B) and A \to B are isomorphic. The usual proof is by parametricity, instead we’ll use a closely related category-theoretic argument: the coYoneda lemma.

+ +

The CoYoneda Lemma

+ +

The coYoneda lemma is a generalization of the equivalence described above. I’ll start with the ordinary version which uses coends and presheaves.

+ +

The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let’s break that down.

+ +

Coends

+ +

A coend is a construction that is very similar to the parametric existential quantifier. If you’re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.

+ +

You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be “irrelevant”.

+ +

So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can “hide the A”. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then

+ +

\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]

+ +

A function parameterized by types like \(pack \) that has this property is called a co-wedge from \(M \).

+ +

A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are universal, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.

+ +

If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all relations implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don’t act like functions.

+ +

Presheaves

+ +

Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of “inputs”, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that “preprocesses” the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.

+ +

So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can’t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).

+ +

Enriched Categories and Enriched CoYoneda

+ +

But there’s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as sets, but we wanted an isomorphism of types in our programming language. We can reconcile this by considering enriched category theory and the enriched coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word “set” with “object of \(V \)”.

+ +

Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) itself as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the internal hom aka exponential.

+ +

Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a contravariant functor. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.

+ +

Now that we’re working all inside our language, let’s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions

+ +
f : (Contravariant Q) => (∃ Γ. (Δ -> Γ) × (Q Γ)) -> Q Δ
+g : (Contravariant Q) => Q Δ -> ∃ Γ. (Δ -> Γ) × (Q Γ)
+ +

If you try to implement them you won’t be able to get it wrong, but here they are:

+ +
f (k, qΓ) = contramap k qΓ
+g qΔ = (id, qΔ)
+ +

where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.

+ +

Closure Version as CoYoneda

+ +

Now it’s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.

+ +

We want to prove the closure conversion isomorphism:

+ +

\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]

+ +

using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)

+ +

\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]

+ +

Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):

+ +

\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]

+ +

Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:

+ +

\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]

+ +

Conclusion

+ +

Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in Haskell and is also central to the Day Convolution, which can be used to give semantics to separation logic.

+ +

Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we’re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.

+ +

You might also like

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/08/29/-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017/index.html b/blog/2017/08/29/-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017/index.html new file mode 100644 index 00000000..fc2cbcaf --- /dev/null +++ b/blog/2017/08/29/-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017/index.html @@ -0,0 +1,169 @@ + + + + + + [Why am I going to ICFP 2017? (cross-post)](https://williamjbowman.com/blog/2017/08/29/why-am-i-going-to-icfp-2017/) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Why am I going to ICFP 2017? (cross-post)

+

+ ::

+

By: William J. Bowman

+
+ +

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/09/25/plt-redex-faq/index.html b/blog/2017/09/25/plt-redex-faq/index.html new file mode 100644 index 00000000..105f7106 --- /dev/null +++ b/blog/2017/09/25/plt-redex-faq/index.html @@ -0,0 +1,649 @@ + + + + + + PLT Redex FAQ + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

PLT Redex FAQ

+

+ :: tutorial, PLT Redex

+

By: Ben Greenman, Sam Caldwell

+
+ +

A short guide to Redex concepts, conventions, and common mistakes.

+ + +
+ +

To contribute to this FAQ, submit issues and pull requests to: https://github.com/nuprl/website/

+ +

Q. What is Redex useful for?

+ +
    +
  1. declaring regular tree grammars
    2. +
  2. defining pattern-based judgments and relations on terms
    3. +
  3. testing properties of the above
+ +

More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.

+ +

Q. What is Redex not useful for?

+ +

Proving theorems about a grammar, judgment, or relation.

+ +

Q. What is a term?

+ +

Informally, a term is:

+ +
    +
  • a Redex “atom”, or
    • +
  • an object that represents a sequence of characters.
+ +

More formally, a term is the result of evaluating (term X), where X is any syntactically-correct Racket expression.

+ +

Examples:

+ +
$ racket
+Welcome to Racket v6.10.0.3.
+> (require redex/reduction-semantics)
+> (term 42)
+42
+> (term (+ 2 2))
+'(+ 2 2)
+> (term ("hello" world (#false)))
+'("hello" world (#f))
+ +

Some terms may look strange. That’s OK, because a term by itself has no meaning.

+ +

Terms can refer to previously-defined values using the unquote escape.

+ +
> (define x (term 42))
+> (term (+ 2 x))
+'(+ 2 x)
+> (term (+ ,x (unquote x)))
+'(+ 42 42)
+ +

Q. What is a Redex model?

+ +

A Redex model is collection of tools for working with terms. The tools may include:

+ +
    +
  • languages, to define a grammar for terms
    • +
  • judgments, to describe properties of terms or relations between terms
    • +
  • metafunctions, to transform terms
    • +
  • reduction relations, to define a term rewriting system
+ +

The goal of these tools is to encode a “real thing” (maybe, a programming language) using Redex terms.

+ +

Q. What is a language?

+ +

A Redex language is a named set of non-terminals, patterns, and binding forms. For example, a Redex model of the natural numbers might start with this language:

+ +
(define-language nat
+  [N ::= Zero
+         (Plus1 N)])
+ +
    +
  • the name of the language is nat
    • +
  • the non-terminal N is associated with two patterns: Zero and (Plus1 N)
    • +
  • there are no binding forms
+ +

Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.

+ +

The non-terminal N describes all terms that are either:

+ +
    +
  1. the symbol Zero
    2. +
  2. lists of the form (Plus1 N), where N is either Zero or another “Plus1”
+ +

For example,

+ +
(term Zero)
+(term (Plus1 Zero))
+(term (Plus1 (Plus1 Zero)))
+(term (Plus1 (Plus1 (Plus1 Zero))))
+;; .... and so on
+ +

If a language has binding forms, then some terms can introduce names. See the question on binding forms (below) for an example.

+ +

Q. What is a pattern?

+ +

A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains named non-terminals from the language, then you can ask whether a Redex term matches the pattern.

+ +

A named non-terminal for a language L is an identifier made of: (1) a non-terminal from L, (2) an underscore (_), and (3) any other identifier. See the FAQ entry below.

+ +

For example, (redex-match? L p t) returns #true if the term t matches the pattern p relative to the language L.

+ +
(define-language nat
+  [N ::= Zero (Plus1 N)])
+
+(redex-match? nat N_some-name (term Zero))
+;; #true
+(redex-match? nat (Plus1 N_a) (term Zero))
+;; #false
+(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero))))
+;; #true
+ +

If (redex-match? L p t) is #true, then (redex-match L p t) shows how named non-terminals in the pattern bind to subterms of t.

+ +
(redex-match nat N_0 (term Zero))
+;; (list (match (list (bind 'N_0 'Zero))))
+(redex-match nat (Plus1 N_0) (term Zero))
+;; #f
+(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero))))
+;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))
+ +

Q. What is a named non-terminal?

+ +

A named non-terminal in a language L is an identifier of the form X_Y, where:

+ +
    +
  • X is a non-terminal from L
    • +
  • Y is any identifier
+ +

The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.

+ +
(define-language trees
+  [binary-tree ::= Leaf
+                   (Node binary-tree binary-tree)])
+
+(redex-match trees
+  (Node binary-tree_left binary-tree_right)
+  (term (Node Leaf (Node Leaf Leaf))))
+;; (list
+;;  (match
+;;   (list (bind 'binary-tree_left 'Leaf)
+;;         (bind 'binary-tree_right '(Node Leaf Leaf)))))
+ +

Q. What else can patterns express?

+ +

Redex patterns may contain special identifiers to guide pattern-matching. For instance:

+ +
    +
  • The _ pattern matches any term (and does not bind).
    • +
  • A pattern (p …) matches any sequence whose elements match the pattern p. If the pattern p is a named non-terminal, then the non-terminal binds a sequence of subterms.
+ +

Examples:

+ +
(redex-match? nat (Plus1 _) (term (Plus1 9)))
+;; #true
+(redex-match? nat (N_0 ...) (term ()))
+;; #true
+(redex-match? nat (N_0 ...) (term (Zero)))
+;; #true
+(redex-match nat (N_0 ...) (term (Zero Zero Zero)))
+;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))
+ +

See the Redex reference for the full pattern language, including the named and unique non-terminals of the form X_!_Y.

+ +

Q. What can patterns not express?

+ +
    +
  • Disjunctions of patterns, e.g., “number or boolean”. (Use a language non-terminal.)
    • +
  • Negations of patterns. (Compose not with redex-match?.)
    • +
  • Some non-regular patterns. (Named dots ..._N or define-language with a side condition may be able to help.)
+ +

Q. What is a judgment?

+ +

A Redex judgment form defines a relation on terms. The relation is defined by a set of inference rules.

+ +

Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:

+ +
    +
  • well-formedness conditions (i.e., whether a term contains free variables)
    • +
  • type checking rules
    • +
  • type inference rules
    • +
  • evaluation relations
+ +

Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.

+ +
(define-language nat
+  [N ::= Zero (Plus1 N)])
+
+(define-judgment-form nat
+  #:mode (N= I I)
+  #:contract (N= N N)
+  [
+   --- Zero=
+   (N= Zero Zero)]
+  [
+   (where (Plus1 N_0--) N_0)
+   (where (Plus1 N_1--) N_1)
+   (N= N_0-- N_1--)
+   --- Plus1=
+   (N= N_0 N_1)])
+ +
    +
  1. the language is nat; Redex uses the language to interpret patterns
    2. +
  2. the mode is (N= I I); this means N= is the name of a judgment that expects two input terms (or, N= is a binary relation on terms)
    3. +
  3. the contract is (N= N N); in other words, N= expects two terms that match the N non-terminal from the nat language
    4. +
  4. there are two inference rules, named Zero= and Plus1=
    5. +
  5. the Zero= rule states that (N= Zero Zero) always holds
    6. +
  6. the Plus1= rule states that (N= N_0 N_1) holds if N_0 and N_1 are both Plus1 terms whose contents are related by N=
+ +

The where clauses are guards. When Redex tries to apply a rule with premises of the form (where pattern term), it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See the Redex reference for more.

+ +
(judgment-holds (N= Zero Zero))
+;; #true
+(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero))))
+;; #true
+(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero))))
+;; #false
+ +

Note: the inference rules form a set, not a sequence. So when you ask Redex whether (judgment-holds (N= Zero Zero)), it applies all rules that match (N= Zero Zero). For N= this is just one rule, but in general it could be many rules.

+ +

Q. What is a judgment form #:mode?

+ +

A #:mode declaration expects a list of the form (id pos-use …), where id is an identifier and each pos-use is either I or O. These declarations say four things:

+ +
    +
  1. id is the name of a new judgment form
    2. +
  2. id expects N arguments, where N is the number of pos-use symbols
    3. +
  3. id expects an input at each position where the mode contains an I
    4. +
  4. id produces an output at each position where the mode contains an O
+ +

For example, a type inference judgment may take an expression as input and output a type. Here’s a fast and easy type inference judgment for arithmetic expressions. Given any term e_0, the judgment outputs the type Int.

+ +
(define-language Arith
+  (e ::= integer (e + e))
+  (τ ::= Int))
+
+(define-judgment-form Arith
+  #:mode (infer-type I O)
+  #:contract (infer-type e τ)
+  [
+   --- T-Int
+   (infer-type e_0 Int)])
+ +

Q. What can judgments not express?

+ +

Redex does not support inference rules that require guessing.

+ +

One example of this is a transitivity rule: "τ_0 is related to τ_2 if there exists a τ_1 such that τ_0 is related to τ_1 and τ_1 is related to τ_2". The following example tries to define a transitive subtyping (<:) relation, but Redex rejects the S-Trans rule.

+ +
(define-language SomeTypes
+  (τ ::= (→ τ τ) Integer))
+
+(define-judgment-form SomeTypes
+  #:mode (<: I I)
+  #:contract (<: τ τ)
+  [
+   (<: τ_0 τ_1)
+   (<: τ_1 τ_2)
+   --- S-Trans
+   (<: τ_0 τ_2)]
+  [
+   --- S-Refl
+   (<: τ_0 τ_0)]
+  [
+   (<: τ_dom-1 τ_dom-0)
+   (<: τ_cod-0 τ_cod-1)
+   --- S-Arrow
+   (<: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])
+ +

The error is that in the rule S-Trans, the named non-terminal τ_1 appears in an input position but is not bound to a term.

+ +

Q. What is a metafunction?

+ +

A metafunction is a term-level function on terms.

+ +

Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.

+ +

Here is a metafunction that returns #true when given two equal natural numbers. The definition is similar to the N= judgment form.

+ +
(define-metafunction nat
+  N=? : N N -> boolean
+  [(N=? Zero Zero)
+   #true]
+  [(N=? N_0 N_1)
+   (N=? N_0-- N_1--)
+   (where (Plus1 N_0--) N_0)
+   (where (Plus1 N_1--) N_1)]
+  [(N=? N_0 N_1)
+   #false])
+ +
    +
  • the metafunction is named N=?
    • +
  • its contract is N N -> boolean, this means N=? expects 2 terms that match the N pattern and returns a term that matches the pattern boolean
    • +
  • there are three cases; the second case is guarded by two where clauses
+ +

Any occurrence of (N=? ….) in any term is evaluated.

+ +
(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero))))
+;; #true
+(term ((N=? Zero Zero) Zero))
+;; '(#true Zero)
+(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero))))
+;; #false
+ +

Any occurrence of N=? outside a term is an error.

+ +

Metafunction where-clauses are analogous to judgment form where-clauses. See the Redex reference for more.

+ +

Note: the cases in a metafunction form a sequence, not a set. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.

+ +

Q. Should I use a metafunction or a judgment form?

+ +

Use a judgment form.

+ +

Metafunctions are handy, but judgments are easier to read and debug and maintain.

+ +

Q. What is a reduction relation?

+ +

A reduction relation is a set of term-rewriting rules.

+ +

Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.

+ +
    +
  • The language tells Redex how to interpret patterns.
    • +
  • The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.
    • +
  • The rewrite rules have the form (—> term term guard …). The term on the left is the input, the term on the right is the output.
+ +

See the Redex reference for a full description of the guards.

+ +

The preferred way to define a reduction relation is to define a language that includes three non-terminals:

+ +
    +
  1. a non-terminal for the domain of the reduction relation
    2. +
  2. a non-terminal for a subset of the domain that cannot reduce further
    3. +
  3. a non-terminal for evaluation contexts
+ +

An evaluation context is a term that contains a hole. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite — in particular, the sub-term that matches the hole.

+ +

In the following example, bexp is the domain of a reduction relation. A bexp term represents a boolean expression, which can be #true or #false or a conjunction of expressions or a disjunction of expressions. The boolean expressions #true and #false are also values (val); these cannot reduce further. The non-terminal E is for evaluation contexts.

+ +
(define-language Bool
+  (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp))
+  (val ::= #true #false)
+  (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E)))
+
+(define step
+  (reduction-relation Bool
+    #:domain bexp
+    [--> (in-hole E (val_lhs ∧ val_rhs))
+         (in-hole E val_new)
+         ∧-step
+         (where val_new ,(and (term val_lhs) (term val_rhs)))]
+    [--> (in-hole E (val_lhs ∨ val_rhs))
+         (in-hole E val_new)
+         ∨-step
+         (where val_new ,(or (term val_lhs) (term val_rhs)))]))
+ +

The function apply-reduction-relation applies a reduction relation to a term and returns a list of ways that the term can step.

+ +
(apply-reduction-relation step (term #true))
+;; '()
+(apply-reduction-relation step (term (#true ∧ #true)))
+;; '(#true)
+(apply-reduction-relation step (term (#true ∧ #false)))
+;; '(#false)
+(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true)))
+;; '((#true ∧ #true))
+ +

Three things about the reduction relation step:

+ +
    +
  1. Using in-hole on the first argument of —> searches a term for a subterm that Redex can apply a reduction rule to.
    2. +
  2. Using in-hole on the second argument of —> puts a new value back into the hole in the evaluation context.
    3. +
  3. The unquote operator (,) escapes to “Racket mode” (see below) to evaluate a conjunction or disjunction.
+ +

A judgment or metafunction is a formal alternative to “escaping to Racket”, but escaping can be convenient.

+ +

Note: the cases in a reduction relation form a set, not a sequence. If more than one case matches, Redex applies them all.

+ +

Q. What is “Racket mode”? What is “Redex mode”?

+ +

Code in a Redex model is sometimes evaluated in “Racket mode” and sometimes evaluated in “Redex mode”. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.

+ +

Key points:

+ +
    +
  1. A Redex program starts in Racket mode.
    2. +
  2. The X in (term X) is evaluated in Redex mode …
    3. +
  3. … unless X contains unquoted sub-expressions. Unquoting escapes to Racket mode …
    4. +
  4. … and terms inside an unquoted sub-expression are evaluated in Redex mode.
+ +

In other words, term enters Redex mode and unquote (,) escapes back to Racket.

+ +

Redex implicitly switches to Redex mode in a few other places, for example:

+ +
    +
  • the right-side of a where clause is in Redex mode
    • +
  • the result of a metafunction is in Redex mode
+ +

When in doubt, try using an unquote. Redex will raise an exception if it finds an unquote in Racket mode.

+ +

Q. Are side-conditions evaluated in “Racket mode” or “Redex mode”?

+ +

A (side-condition e) sometimes evaluates e as a Racket expression and sometimes evaluates e as a Redex expression.

+ +
    +
  • reduction relations and metafunctions expect a Racket expression
    • +
  • judgments expect a Redex expression
+ +

Q. What is a binding form?

+ +

In the lambda calculus, λ-terms bind variables. A term (λ x M) means that any free occurrence of x in the sub-term M refers to the x from the λ-term.

+ +

Redex can express this idea with a binding form.

+ +
(define-language Λ
+  [e ::= (e e) x (λ x e)]
+  [x ::= variable-not-otherwise-mentioned]
+  #:binding-forms
+  (λ x_0 e_0 #:refers-to x_0))
+ +

Note: all the non-terminals in a language must be defined before the #:binding-forms keyword. If a non-terminal definition appears after the #:binding-forms keyword, then Redex will interpret the “definition” as a binding form.

+ +

Binding forms work together with Redex’s functions for substitution and alphabetic equivalence.

+ +
(alpha-equivalent? Λ
+  (term (λ x x))
+  (term (λ y y))))
+;; #true
+
+(define-metafunction Λ
+  test-substitute : e -> e
+  [(test-substitute (λ x_0 e_0))
+   (substitute e_0 x_0 y)])
+(term (test-substitute (λ z (z z))))
+;; '(y y)
+ +

Q. What is …? What is ….?

+ +

Three dots (…) is for building patterns. If p is a pattern then (p …) matches any list whose elements all match p.

+ +
(define-language L)
+(redex-match? L (number ... boolean ...) (term (1 2 #true #true)))
+;; #true
+ +

Four dots (….) may be used in define-extended-language to extend a previously-defined non-terminal.

+ +
(define-language C
+  (keyword ::= auto break case))
+(define-extended-language C++
+  C
+  (keyword ::= .... class))
+
+(redex-match? C keyword (term auto))
+;; #true
+(redex-match? C keyword (term class))
+;; #false
+(redex-match? C++ keyword (term auto))
+;; #true
+(redex-match? C++ keyword (term class))
+;; #true
+ +

Q. Where to learn more about Redex?

+ +

“Critical path” resources:

+ + + +

“Procrastination” resources:

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/index.html b/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/index.html new file mode 100644 index 00000000..72f14eee --- /dev/null +++ b/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/index.html @@ -0,0 +1,438 @@ + + + + + + Final Algebra Semantics is Observational Equivalence + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Final Algebra Semantics is Observational Equivalence

+

+ :: category theory, math, final encoding, observational equivalence

+

By: Max New

+
+ +

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

+ +

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

+ + +

Intuition: Implementing a Signature

+ +

As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.

+ +

We could specify such a datatype by specifying a little language of numbers and finite multisets. We’ll have two “sorts” num and multiset, a constant for every integer, and an addition function

+ +
'n : () -> num;
+add : (num, num) -> num
+ +

subject to the silly-looking equation:

+ +
add('n,'m) = '(n + m)
+ +

and some operations on multisets

+ +
empty : () -> multiset;
+singleton : (num) -> multiset;
+union : (multiset, multiset) -> multiset;
+remove : (num, multiset) -> multiset;
+count : (num, multiset) -> num
+ +

subject to the computational equations:

+ +
count('n, empty) = '0
+count('n, singleton('n)) = '1
+count('n, singleton('m)) = '0
+count('n, union(s,t)) = add(count('n,s), count('n, t))
+count('n, remove('n,s)) = '0
+count('n, remove('m,s)) = count('n,s)
+ +

These are “all” of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively want for reasoning about our programs. For instance, clearly union should be commutative, and remove should be idempotent, but it’s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the “initial algebra”. In Haskell, we could say

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
 +
+ 
data MultiSet = Empty 
+  | Singleton Integer
+  | Union MultiSet MultiSet
+  | Remove Integer MultiSet
+  deriving (Eq)
+
+count :: Integer -> MultiSet -> Integer
+count n Empty = 0
+count n (Singleton m) | n == m = 1
+count n (Singleton m) | n /= m = 0
+count n (Union s t) = (count n s) + (count n t)
+count n (Remove m s) | n == m = 0
+count n (Remove m s) | n /= m = count n s
+
+
+
+ +

Then it is completely obvious that all of our equations hold, but then Union is not commutative, as ghci will tell us:

+ +
+ + + + +
+
+ 
1
+2
 +
+ 
> (Singleton 1 `Union` Singleton 2) == (Singleton 2 `Union` Singleton 1) 
+False
+
+
+
+ +

However, there is another encoding that will give us that union is commutative and remove n is idempotent and actually every equation we could possibly want! It’s called the “final encoding” or “final algebra”. In Haskell, this looks like:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
+24
 +
+ 
data MultiSet' = MultiSet' { _count :: Integer -> Integer }
+
+count' :: Integer -> MultiSet' -> Integer
+count' n m = _count m n
+
+empty :: MultiSet'
+empty = MultiSet' { _count = \n -> 0 }
+
+singleton :: Integer -> MultiSet'
+singleton n = MultiSet' { _count = \m -> if n == m
+                                         then 1
+                                         else 0 }
+
+union :: MultiSet' -> MultiSet' -> MultiSet'
+union s t = MultiSet' { _count = \n -> (count' n s) + (count' n t) }
+
+remove :: Integer -> MultiSet' -> MultiSet'
+remove n s = MultiSet' { _count = \m -> if n == m
+                                        then 0
+                                        else count' n s }
+
+test' = and [ count' n s == count' n t | n <- [0..1000]]
+s = singleton 1 `union` singleton 2
+t = singleton 2 `union` singleton 1
+
+
+
+ +

Now we can verify that union is commutative because

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
union s t = MultiSet' { _count = \n -> (count' n s) + (count' n t) }
+          = MultiSet' { _count = \n -> (count' n t) + (count' n s) }
+		  = union t s
+
+
+
+ +

since + is commutative. Equality isn’t decidable anymore so I can’t give you a simple piece of code to witness this, but we can test our example before and we won’t be able to distinguish them, no surprise:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
 +
+ 
> let s = singleton 1 `union` singleton 2
+> let t = singleton 2 `union` singleton 1
+> and [ count' n s == count' n t | n <- [0..1000]]
+True
+
+
+
+ +

How do we know this is the “best” or at least “most canonical” implementation of our datatype? The intuition is that we really don’t care at all how our multisets are implemented as long as they behave the right way with respect to count since count returns an Integer, a type we do understand. Our encoding accomplishes this by representing a multiset s by the partially applied function \n -> count n s.

+ +

The formal name for this idea is observational equivalence. We say that two closed terms s,t of sort multiset are observationally equivalent if for any term C of type num that has s as a subterm, we can swap t in for s and prove that the two terms are equal. For instance C might be count(3, union(s, singleton(3))) or add(4,remove(5,s)). Then we’ve reduced the possibly complicated equality for multiset to the simple equality of num.

+ +

Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see here), but let’s see what all this talk about “algebras”, initial or final is all about.

+ +

Formalization Attempt 1: Algebras of a Theory

+ +

First, our little language of numbers and multisets is called a theory. The specific category gadget that we’ll use to describe it is a multi-sorted Lawvere theory, or just Lawvere theory for short.

+ +

A Lawvere theory is a category with finite products all of whose objects are finite products of a collection of sorts \(S\). We can construct this category from our little language above by making the objects be contexts \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) modulo the equations we’ve specified. We’ll use the letter \(T\) to mean a Lawvere theory.

+ +

Then a \(T\)-algebra is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).

+ +

Finally a morphism of \(T\)-algebras from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.

+ +

Then we have a category we’ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them always exist.

+ +

The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don’t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of “initial algebra” before, except that this time we defined count as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for count.

+ +

However, an egregious flaw presents itself when we look at what the final algebra is. It’s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.

+ +

Formalization: Algebras of a Theory Extension

+ +

To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts num and multiset in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know exactly what numbers are!

+ +

To formalize this we’ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We’ll formalize this as a morphism of theories. A morphism of theories is a strict product-preserving functor from one theory to another. The strictness ensures that we don’t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it’s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don’t want to consider all such morphisms, only the ones that are “conservative extensions”, which means

+ +
    +
  1. there are no new closed terms at old types
    2. +
  2. closed terms that were different before remain different.
+ +

In our example (1) ensures that we don’t add any new exotic numbers like undefined or ∞, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).

+ +

We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a functor on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).

+ +

Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch’s paper, but there it is separated into two properties fullness and faithfulness, and doesn’t use the initial algebras and \(m^*\) explicitly.

+ +

Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.

+ +

Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that

+ +
    +
  1. The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.
    2. +
  2. The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.
+ +

The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a congruence relation on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called adequacy.

+ +

Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a stronger congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:

+ +
+

Strongest adequate congruence relation

+ +

which is exactly the definition of observational equivalence you will find in, say Pitt’s chapter of Advanced TAPL. There is a difference in the meaning of adequacy, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it’s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that adequacy is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.

+ +

Conclusion: Tying it back to Final Encodings

+ +

So now we’ve seen that

+ +
+

Final algebras are equivalent to initial algebras modulo observational equivalence

+ +

Of course we haven’t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn’t use very many properties of sets in our definition, so it’s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.

+ +

Another question is how to implement this when we have a proper type theory and not just some boring sorts. In particular, if we have function types, then we won’t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/index.html b/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/index.html new file mode 100644 index 00000000..8205219d --- /dev/null +++ b/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/index.html @@ -0,0 +1,297 @@ + + + + + + Monotonicity Types: Towards A Type System for Eventual Consistency + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Monotonicity Types: Towards A Type System for Eventual Consistency

+

+ :: types, monotonicity, CRDTs, eventual consistency

+

By: Kevin Clancy

+
+ +

A few weeks back, we published a draft of an article entitled Monotonicity Types. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.

+ + +

But first, what, precisely, do we mean by monotonicity? Here’s a short definition:

+ +

A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:

+ +
    +
  1. \(p \leq p\) (reflexivity)
    2. +
  2. \(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)
    3. +
  3. \(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)
+ +

If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is monotone if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).

+ +

So, said another way, increasing the input to a monotone function causes an increase to its output.

+ +

Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [Meiklejohn et al. (2015)] and BloomL [Conway et al. (2012)], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [Shapiro et al. (2011)] and LVars [Kuper et al. (2013)] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.

+ +

Thus if a user would like to make use of such a language for concurrent and distributed programming, they’re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.

+ +

To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let’s look at an example. Suppose we need a function to determine whether a replicated counter’s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter’s value:

+ +
fun IsOdd(x : Nat) = x % 2 == 1
+ +

However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can’t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.

+ +
fun MoreThanTen(x : Nat) = x > 10
+ +

The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x > 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P’s value is greater than 10, we don’t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P’s current value. This is where MoreThanTen’s monotonicity becomes useful. At the point when all replicas have received P’s current information, every replica in the system will agree that MoreThanTen applied to the counter’s value returns true.

+ +

We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.

+ +

Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an sfun, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.

+ +

Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone separately in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).

+ +

The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:

+ +

1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)

+ +

2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)

+ +

An sfun type, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also qualifies each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).

+ +

Terms outside of sfun abstractions are typed using a global typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.

+ +

\( \Gamma \vdash t : T \)

+ +

A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:

+ +

\( \Gamma;\Omega;\Phi \vdash t : T \)

+ +

Here the global type environment \( \Gamma \) contains all of the variables bound outside of the sfun, the ambient type environment \( \Omega \) contains the list of the sfun’s formal arguments, and the lifted type environment \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let’s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.

+ +

$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$

+ +

Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction’s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a “slot” named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction’s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.

+ +

The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction’s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function after it has been applied. Unfortunately, this isn’t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, before it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun’s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called ambient maps. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).

+ +

The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise’s derivation, but a full treatment is outside the scope of this post.

+ +

Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)’s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).

+ +

We’re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or “computation model”) which can be viewed in one of two ways:

+ +
    +
  1. The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.
    2. +
  2. The composition of an sfun’s components, before the sfun is ever applied.
+ +

Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications—specifically two applications in the case of monotonicity. If the lifted reduction of an sfun’s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun’s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It’s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.

+ +

Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn’t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the “lifted reduction step” \( f(g(v)) \to (f \circ g) (v) \).

+ +

We’ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn’t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:

+ +
    +
  1. X increments, resulting in a count of 1
    2. +
  2. X sends a synchronization message to Y, containing X’s count 1
    3. +
  3. X receives a synchronization message from Y containing a count of 1
+ +

At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map’s finite representation maps to 0. Such counters are called GCounters, and described in detail by [Shapiro et al. (2011)].

+ +

GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).

+ +

[Meiklejohn et al. (2015)] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and > are standard arithmetic operators for working with natural numbers.

+ +
getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k]
+joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n]
+span :: (m:NatMap) ⇒ Nat[↑ m]
+max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b]
+emptyMap :: NatMap
++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y]
+> :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y]
+
+type GCounter = { map : NatMap }
+
+sfun sumCounters(x : GCounter, y : GCounter) 
+ : GCounter[↑ x, ↑ y] =
+ let xMap : NatMap[↑ x, ↑ y] = x.map
+ let yMap : NatMap[↑ x, ↑ y] = y.map
+ let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap)
+ fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) 
+  : NatMap[↑ x, ↑ y] =
+  let cond : Bool[↑ x, ↓ y] = k > maxSpan
+   if cond then
+    acc
+   else
+    let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k))
+    sumCell (k+1) acc'
+ let initMap : IntArray[↑ x, ↑ y] = emptyMap
+ GCounter { map = sumCell 0 initMap }
+ +

While our system can handle much of this example, it can’t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [Vazou et al. (2014)]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.

+ +

So there’s still much to do! If you’re interested in more details behind the type system, have a look at Kevin’s blog article, Monotonicity Through Types, or have a look at the full Monotonicity Types preprint for more.

+ +

References

+ +

C. Meiklejohn and P. Van Roy. Lasp: A language for distributed, coordination-free programming. In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.

+ +

N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. Logic and lattices for distributed programming. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.

+ +

M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. Conflict-Free replicated data types. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.

+ +

L. Kuper and R. R. Newton. LVars: Lattice-based data structures for deterministic parallelism. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.

+ +

N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. Refinement types for Haskell. SIGPLAN Not. 49, 9 (August 2014), 269–282.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/01/17/-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html/index.html b/blog/2018/01/17/-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html/index.html new file mode 100644 index 00000000..6ebb1fd6 --- /dev/null +++ b/blog/2018/01/17/-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html/index.html @@ -0,0 +1,169 @@ + + + + + + [How to prove a compiler correct (cross-post)](https://dbp.io/essays/2018-01-16-how-to-prove-a-compiler-correct.html) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

How to prove a compiler correct (cross-post)

+

+ ::

+

By: Daniel Patterson

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/01/19/-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist/index.html b/blog/2018/01/19/-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist/index.html new file mode 100644 index 00000000..82d8d04f --- /dev/null +++ b/blog/2018/01/19/-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist/index.html @@ -0,0 +1,169 @@ + + + + + + [Untyped Programs Don't Exist (cross-post)](https://williamjbowman.com/blog/2018/01/19/untyped-programs-don-t-exist/) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Untyped Programs Don’t Exist (cross-post)

+

+ ::

+

By: William J. Bowman

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/index.html b/blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/index.html new file mode 100644 index 00000000..d852ffe7 --- /dev/null +++ b/blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/index.html @@ -0,0 +1,169 @@ + + + + + + [Making an IDE Plugin for DrRacket (cross-post)](https://lang.video/blog/2018/03/21/making-an-ide-plugin-for-drracket/) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Making an IDE Plugin for DrRacket (cross-post)

+

+ :: tutorials

+

By: Leif Andersen

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/04/23/-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html/index.html b/blog/2018/04/23/-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html/index.html new file mode 100644 index 00000000..2aa92c7f --- /dev/null +++ b/blog/2018/04/23/-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html/index.html @@ -0,0 +1,169 @@ + + + + + + [How to prove a compiler fully abstract (cross-post)](https://dbp.io/essays/2018-04-19-how-to-prove-a-compiler-fully-abstract.html) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

How to prove a compiler fully abstract (cross-post)

+

+ ::

+

By: Daniel Patterson

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/04/27/the-racket-school-2018-create-your-own-language/index.html b/blog/2018/04/27/the-racket-school-2018-create-your-own-language/index.html new file mode 100644 index 00000000..a5f909e6 --- /dev/null +++ b/blog/2018/04/27/the-racket-school-2018-create-your-own-language/index.html @@ -0,0 +1,192 @@ + + + + + + The Racket School 2018: Create your own language + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

The Racket School 2018: Create your own language

+

+ :: event, Racket

+

By: Ben Greenman

+
+ +

The Racket School 2018: Create your own language • 9–13 July • Salt Lake City

+ + +

The Racket team has spent over thirty years developing and refining a coherent intellectual tradition for studying and building programming languages. This year’s school will introduce participants to Racket’s framework for language-oriented programming, which the summer school faculty recently spelled out in a a cover article in the Communications of the ACM

+ +

Concretely, the 2018 Racket Summer School will cover the following topics:

+ +
    +
  • the spectrum of programming languages;
    • +
  • modules and syntax, or languages as libraries;
    • +
  • DrRacket’s support for language-oriented programming;
    • +
  • a domain-specific language for adding types to languages;
    • +
  • tools and techniques for implementing notational conveniences; and
    • +
  • research challenges in language-oriented programming.
+ +

If these topics intrigue you, attend the Racket Summer School:

+ + + +

This is not your run-of-the-mill summer school. We will do our best to make it exciting, entertaining, and useful to a broad spectrum of attendees, both academic and industrial.

+ +

P.S. We will send you your first problem set in June, a month before the summer school to whet your appetite.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/05/08/sampling-gradual-typing-performance/index.html b/blog/2018/05/08/sampling-gradual-typing-performance/index.html new file mode 100644 index 00000000..1741b4e8 --- /dev/null +++ b/blog/2018/05/08/sampling-gradual-typing-performance/index.html @@ -0,0 +1,284 @@ + + + + + + Sampling Gradual Typing Performance + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Sampling Gradual Typing Performance

+

+ :: gradual typing, migratory typing, performance, statistics, Takikawa constant

+

By: Ben Greenman, Zeina Migeed

+
+ +

This post explains the sampling method introduced in the paper On the Cost of Type-Tag Soundness

+ + +

Quick Reference: How to apply the method

+ +
    +
  1. Find an untyped program, measure its running time.
    2. +
  2. Define a granularity for type annotations (by-function, by-module, by-program, ….).
    3. +
  3. Define a sample size s and number of samples r.
    4. +
  4. Randomly select s configurations uniformly at random, measure their running time.
    5. +
  5. Repeat the previous step r times.
    6. +
  6. Pick a positive real number D.
    7. +
  7. Count the proportion of configurations in each sample with running time less-than-or-equal-to D
    8. +
  8. Build a 95% confidence interval for the r proportions computed in the previous step
    9. +
  9. Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to D
+ +

Background: what to measure

+ +

A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for “adding static typing” has a few steps:

+ +
    +
  • add a syntax for type annotations
    • +
  • add a static type checker
    • +
  • add a semantics for statically-typed parts of the program
+ +

If the semantics for statically-typed parts of the program is not the same as the semantics for dynamically-typed parts, then it is important to measure performance.

+ +

The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.

+ +

An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?

+ +

The next two sections give two methods for answering this question.

+ +

Exhaustive Method

+ +

One way to answer our easier question is to remove type annotations one “unit” at a time and measure the running time of all these partially-typed programs. We call the “unit” the granularity of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the “partially-typed programs” the configurations of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity — I can’t just use the word configurations without telling you the granularity I have in mind.

+ +

After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number D and count the number of configurations that run at most D times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if you are willing to accept at most a Dx slowdown, and you add your own type annotations to your own program, then there’s some hope that your configuration runs at most D times slower than your original program.

+ +
+

Credit for the exhaustive method: Is Sound Gradual Typing Dead? and Toward Practical Gradual Typing

+ +

Simple Random Approximation Method

+ +

The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.

+ +

On the Cost of Type-Tag Soundness, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most D times slower than the original dynamically-typed program, we:

+ +
    +
  • pick a sample size s (in the paper, we used s = 10M where M is the number of units),
    • +
  • pick a number of samples r (in the paper, we used r = 10),
    • +
  • and build a 95% confidence interval for the true proportion of configurations that run at most D times slower than the original program (from the r proportions of configurations that run at most D times slower than the original program in each of the r samples).
+ +

The method is outlined above, described in the paper, and validated in that paper’s appendix. Please let us know if you have more questions.

+ +
+

Maybe you’re wondering, “gee why do they keep writing out ‘configurations that run at most ….’ instead of something shorter?”. Well, the short version is D-deliverable and it was introduced in the Is Sound Gradual Typing Dead? paper. Unfortunately, (1) that paper instantiated D to 3-deliverable in order to explain a few graphs and (2) at least two published papers (paper 1, paper 2) now cite us as saying 3x overhead is the cutoff between a good migratory typing system and a bad one.

+

…

+

If we can’t trust scientists to understand, then we definitely can’t trust you folks on the internet.

+ +

FAQ

+ +

Q. What is the sampling method useful for?

+ +
    +
  • Making a confidence interval for the true proportion of configurations that run at most D times slower than some baseline, for your favorite value of D.
+ +

Q. What is the sampling method not useful for?

+ +
    +
  • Finding the slowest configuration.
    • +
  • Finding the average running time of all configurations.
    • +
  • Evaluations where “removing types” might involve changing List[Int] to List[Dyn], etc.
    • +
  • Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random
    • +
  • …. many more ….
+ +

Q. Why is it okay to choose D after collecting the samples?

+ +

The “quick reference” at the top of this post suggests choosing a value for D (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of D affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn’t and value that affects the bottom line be fixed before taking samples? (To avoid accidental data dredging.)

+ +

The reason it is ok to pick D after taking the sample is that the running times in the sample are independent of the choice of D.

+ +

For example, if one person chose D=3 and a second person chose D=9, both would follow the same protocol independent of D to take samples.

+ +

Q. How does migratory typing relate to gradual typing?

+ +

Gradual typing is not just about adding a type system to an existing programming language. See Refined Criteria for Gradual Typing and Migratory Typing: 10 Years Later for details.

+ +

Q. Do you have code I can use to plot sampling data?

+ +

Yes, start here:

+ + + +

Please ask questions and open issues if you have trouble. The source is here:

+ + + +

Q. Where is code for the sampling paper?

+ +

Start here:

+ + + +

Source is here:

+ + + +

Closing Thoughts

+ +

Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html b/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html new file mode 100644 index 00000000..cd1d000e --- /dev/null +++ b/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html @@ -0,0 +1,226 @@ + + + + + + A Spectrum of Type Soundness and Performance + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

A Spectrum of Type Soundness and Performance

+

+ :: migratory typing, gradual typing, extended abstract

+

By: Ben Greenman

+
+ +

The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.

+ + +
+

Note: this post is an extended abstract for the paper A Spectrum of Type Soundness and Performance by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018

+ +

A dynamically-typed language runs any program that “looks good” (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both “looks good” and is well-typed according to a type checker.

+ +

A mixed-typed language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.

+ +
Figure 1: Some mixed-typed languages +

Figure 1: Some mixed-typed languages

+ +

These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:

+ +
    +
  • MACLISP defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the Moonual. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).
    • +
  • Strongtalk includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the Pluggable Type Systems position paper.
    • +
  • Typed Racket lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.
    • +
  • Thorn requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.
    • +
  • Reticulated lets a program combine static and dynamic expressions and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek POPL 2017).
+ +

That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system’s protocol for checking dynamically-typed values at a type boundary (between static and dynamic code).

+ +

Main Contribution

+ +

In the paper A Spectrum of Type Soundness and Performance, we define a tiny mixed-typed language and show three ways to define the behavior of programs — based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.

+ +

The three behaviors are inspired by existing languages. A higher order behavior ensures that dynamically-typed values match the static type at a boundary — by checking the value when possible, and by monitoring the value’s future interactions when necessary. A first order behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An erasure behavior does no checking whatsoever.

+ +
+

Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function f, then one way to enforce the type boundary is to wrap f in a proxy to assert that every future call to f returns a number. In this case, the proxy monitors the behavior of f.

+ +

Concretely, the paper defines three formal semantics with the same names. The higher-order semantics enforces full types at the boundaries (Section 2.3). The first-order semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every “selector” operation in typed code, e.g., function application, data structure access (Section 2.5). The erasure semantics simply ignores the types (Section 2.4).

+ +

Each semantics satisfies a different notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the online supplement gives full proofs.

+ +

The paper has more to say about the meta-theory. See section 2 and section 4.

+ +
+

To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for subset of the language (in the spirit of soundiness) or use the name “type soundness” to describe a different property.

+ +

Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations — similar to Typed Racket’s no-check language.

+ +

Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (POPL 2016). The programs we measured are the non-object-oriented ones from our benchmark suite.

+ +

To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises — see section 3 of the paper, section B of the supplement, and/or the slides.

+ +

Implications

+ +
    +
  1. The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the supplement, or slide 114.
    2. +
  2. Programmers using mixed-typed languages need to know what guarantees their types provide. (It is not safe to assume that TypeScript types give the same guarantees as OCaml types!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.
    3. +
  3. The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?
+ +
+

Note: in this post, a mixed-typed language is one that allows any combination of static and dynamic typing. A gradually-typed language is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (SNAPL 2015).

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/index.html b/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/index.html new file mode 100644 index 00000000..bd8d7ca2 --- /dev/null +++ b/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/index.html @@ -0,0 +1,886 @@ + + + + + + Defining Local Bindings in Turnstile Languages + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Defining Local Bindings in Turnstile Languages

+

+ :: turnstile, tutorial, language, dsl

+

By: Sam Caldwell

+
+ +

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

+ +

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

+ +

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

+ +

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

+ + +

The code for this blog post can be found in this gist. To run it, you will need the Turnstile package, which can be installed with raco pkg install +turnstile.

+ +

Turnstile: Typechecking Intertwined with Elaboration

+ +

Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.

+ +

While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.

+ +

Type & Elaboration Rules

+ +

To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.

+ +

Functions

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
(define-type-constructor → #:arity >= 1)
+(define-typed-syntax (λ ([x:id (~datum :) τ_in:type] ...) e) ≫
+  [[x ≫ x- : τ_in.norm] ... ⊢ e ≫ e- ⇒ τ_out]
+  -------------------------------------------------
+  [⊢ (#%plain-lambda- (x- ...) e-) ⇒ (→ τ_in.norm ... τ_out)])
+
+
+
+ +

Looking at this item by item, we see:

+ +
    +
  1. define-type-constructor creates a new type. Here, we say the → requires at least one parameter.
    2. +
  2. define-typed-syntax is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.
    3. +
  3. The next part is a syntax-pattern describing the the shape of the syntax this rule applies to. In this case, we’re defining λ as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body e. The type syntax class is provided by Turnstile, and describes the surface syntax of types (such as those created with define-type-constructor); internal operations over types use the expanded version of the type, which is accessed via the norm attribute.
    4. +
  4. The chevron ≫ on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.
    5. +
  5. The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.
    6. +
  6. Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (⊢). For each formal x, this lets us know what type x has when we find a reference to it in e. In this rule, we are saying “while checking the right-hand-side, assume x—which elaborates to x-—has type τ_in, for each triple in the input syntax (signified by the ellipses ...)”. More on the “elaborates to x-” below.
    7. +
  7. To the right of the turnstile, we write the expression we are checking, e, and patterns e- and τ_out matching the elaboration of e and its type, respectively.
    8. +
  8. After the dashes comes the conclusion, which begins with ⊢. The next part specifies the elaboration of the term. Here, the meaning of the typed λ is given in terms of Racket’s #%plain-lambda. Turnstile uses the convention of a - suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using postfix-in, such as in (require (postfix-in - racket/base)) to bind #%plain-lambda-.
    9. +
  9. Finally, we give the type of the term to the right of the ⇒, referring to pattern variables bound in the premises.
+ +

Renaming Typed Variables

+ +

Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier x with type τ, Turnstile binds x to a macro that knows τ when it expands. References to x now become references to that macro, and expanding them provides access to τ. Concretely, the underlying Racket code implementing this behavior looks roughly like this:

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
(let ([x- (assign-type (generate-temporary #'x) #'τ)])
+  (let-syntax ([x (make-rename-transformer x-)])
+    ... expand and check forms that may reference x ...))
+
+
+
+ +

The type τ is attached as metadata for a new identifier x-, which is what x will transform to at any reference site. In order for this to work, x- must be distinct from x—hence the generate-temporary—to avoid an infinite expansion loop.

+ +

Application

+ +

We can define a version of #%app that type checks function applications to accompany our typed λ:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
 +
+ 
(define-typed-syntax (#%app e_fn e_arg ...) ≫
+  [⊢ e_fn ≫ e_fn- ⇒ (~→ τ_in ... τ_out)]
+  #:fail-unless (stx-length=? #'[τ_in ...] #'[e_arg ...])
+                (num-args-fail-msg #'e_fn #'[τ_in ...] #'[e_arg ...])
+  [⊢ e_arg ≫ e_arg- ⇐ τ_in] ...
+  --------
+  [⊢ (#%plain-app- e_fn- e_arg- ...) ⇒ τ_out])
+
+
+
+ +
    +
  1. The syntax pattern on the first line describes the shape of applications.
    2. +
  2. On the second line, we pattern match the result of expanding and checking e_fn, checking that it produces an arrow type. More specifically, when we defined the arrow type → above, define-type-constructor also implicitly defined a pattern expander ~→ (which uses the Racket ~ prefix convention for syntax patterns) that matches instances of the type.
    3. +
  3. The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.
    4. +
  4. Line 5 checks that each argument expression has the required type. Turnstile uses bidirectional typechecking rules, which either infer the type of a term or checks that a term satisfies a given type. We write ⇐ τ_in in the premise to switch to checking mode.
    5. +
  5. Finally, typed function application elaborates to Racket’s function application, #%plain-app, with the usual suffix, and produces type τ_out for the application
+ +

We can try out these new typed forms on a few examples:

+ +
    +
  • ((λ ([x : Int]) (+ x 1)) 2) successfully typechecks and yields 3.
    • +
  • ((λ ([x : Int]) (+ x 1))) raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."
    • +
  • ((λ ([x : (→ Int Int)]) (x 1)) 2) raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.
+ +

Extending Our Language with Local Bindings

+ +

When writing functional programs, we often want to name various sub-computations. One way to do that is with a let construct, which Turnstile allows us to easily create:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
(define-typed-syntax (let ([x:id e-x] ...) e-body) ≫
+  [⊢ e-x ≫ e-x- ⇒ τ-x] ...
+  [[x ≫ x- : τ-x] ... ⊢ e-body ≫ e-body- ⇒ τ-body]
+  -------------------------------------
+  [⊢ (let- ([x- e-x-] ...) e-body-) ⇒ τ-body])
+
+
+
+ +

Unsurprisingly, this looks very similar to the definition of λ above. Now we can write functions with named intermediate results:

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
(λ ([x : Int])
+  (let ([almost-there (+ x 1)])
+    (+ almost-there 1)))
+
+
+
+ +

However, in Racket it’s common to name such intermediate results using define rather than let. In fact, it’s prescribed by the style guide. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
(λ ([x : Int])
+  (define almost-there (+ x 1))
+  (+ almost-there 1))
+
+
+
+ +

Unfortunately, this is not nearly as easy to do in Turnstile as let.

+ +

Sequences

+ +

At first glance, the issue seems to be that the definition of λ above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of λ to allow its body to be a sequence.

+ +

The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like λ:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
(define-typed-syntax (begin e ...+) ≫
+  [⊢ e ≫ e- ⇒ τ] ...
+  #:with τ-final (stx-last #'(τ ...))
+  -----------------------------------
+  [⊢ (begin- e- ...) ⇒ τ-final])
+
+
+
+ +

This directs type checking to:

+ +
    +
  1. Check each e in the sequence individually, obtaining an expanded e- and inferred type τ for each.
    2. +
  2. Take the last type in the sequence and call it τ-final; Turnstile allows using syntax-parse directives such as #:with as premises.
    3. +
  3. Expand to Racket’s begin (with the usual - suffix) and give the whole expression the type of the last term in the body.
+ +

Now, we can use begin in a revised definition of λ. The new rule takes a non-empty sequence of forms in the body and wraps them in our new begin form for typechecking.

+ +
+ + + + +
+
+ 
1
+2
+3
+4
 +
+ 
(define-typed-syntax (λ ([x:id (~datum :) τ_in:type] ...) e ...+) ≫
+  [[x ≫ x- : τ_in.norm] ... ⊢ (begin e ...) ≫ e- ⇒ τ_out]
+  -------------------------------------------------
+  [⊢ (#%plain-lambda- (x- ...) e-) ⇒ (→ τ_in.norm ... τ_out)])
+
+
+
+ +

Now we need a way to include definitions in these sequences and we’re set!

+ +

The Difficulty With Define

+ +

If we think about how type information is communicated between a binder and its reference we can see why define is a different beast than let

+ +
(let ([x 5]) (+ x 1))
+       ^        ^
+       |        |- TO HERE
+FROM HERE
+ +

When the rule for our let is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the ⊢, as in our rules for λ and let above.

+ +

Define is different.

+ +
(define x 5)
+        ^
+        |------ TO WHERE?
+FROM HERE
+ +

The problem is apparent: we can’t see where the reference to x occurs! The information about the binding needs to escape from the define to the surrounding context. In other words, when we implement define, we don’t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the x binding and its type to the surrounding context.

+ +

Above, in the subsection on “Renaming Typed Variables”, we saw that the context in Turnstile type rules is implemented as syntax transformers with let-like scope (created with let-syntax). One idea would be to mimic this approach, but instead of using let-syntax to achieve let-like scope, use define-syntax to achieve define-like scope.

+ +

Fortunately for us, someone has already tried their hand at writing a define form for Turnstile languages using a define-syntax rename, found in the Turnstile examples. We can take that as our starting point:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
 +
+ 
(define-base-type Void)
+(define- a-deep-dark-void (#%app- void-))
+(define-typed-syntax (define x:id e) ≫
+  [⊢ e ≫ e- ⇒ τ]
+  #:with x- (assign-type (generate-temporary #'x) #'τ #:wrap? #f)
+  -----------------------------------------------------
+  [⊢ (begin-
+       (define-syntax x (make-variable-like-transformer #'x-))
+       (define- x- e-)
+       a-deep-dark-void)
+     ⇒ Void])
+
+
+
+ +

Let’s break it down.

+ +
    +
  1. Create a new type, Void, to assign definitions.
    2. +
  2. Create a constant to serve as the canonical value of type Void.
    3. +
  3. Define a new typed form, define, used as in (define x e).
    4. +
  4. Check the type of the expression e, getting its expansion e- and type τ.
    5. +
  5. Create a new name, x-, and attach the type τ as metadata.
    6. +
  6. Expand to Racket’s begin. Unlike let, begin does not create a new scope; definitions inside a begin are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.
    7. +
  7. Create a macro binding for x that rewrites to x-. By using a define-like form, the macro has the same scoping rules as define, so it will apply to references to x in the surrounding context—exactly what we want. (We are using make-variable-like-transformer to avoid the special treatment the expander gives to rename-transformers. The specifics are beyond the scope of this post.)
    8. +
  8. Define x- to refer to the supplied expression. Note that here define- is Racket’s define.
    9. +
  9. Keep the result of evaluating this form in line with the type by yielding a value of type Void.
+ +

This implementation of define gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
 +
+ 
;; module top level
+(define x 5)
+(+ x 1)
+;;=> 6
+
+
+
+ +

Unfortunately, we encounter a problem if we try to create local definitions:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
(define add2
+  (λ ([x : Int])
+     (define almost (+ x 1))
+     (+ almost 1)))
+;;==> almost: unbound identifier...
+
+
+
+ +

Pointing to the reference on the final line. The problem is that our define and begin forms are not interacting in the way we might have hoped.

+ +

When we expand the body of the function above, we associate x with type Int then start checking the body, wrapped in a begin:

+ +
+ + + + +
+
+ 
1
+2
+3
 +
+ 
(begin
+  (define almost (+ x 1))
+  (+ almost 1))
+
+
+
+ +

Consulting the definition of begin, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of define, yielding

+ +
+ + + + +
+
+ 
1
+2
+3
+4
 +
+ 
(begin-
+  (define-syntax almost ...)
+  (define- almost- ...)
+  a-deep-dark-void)
+
+
+
+ +

Crucially, the expansion of our define form stops at this point, without examining the begin- form and its contained definitions. The interface through which Turnstile invokes the macro expander, local-expand, takes a parameter referred to as the stop list for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.

+ +

The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named erased, that serves (only) to orchestrate stopping expansion. So, the output of our define form actually looks like

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
(erased
+  (begin-
+    (define-syntax almost ...)
+    (define- almost- ...)
+    a-deep-dark-void))
+
+
+
+ +

And since Turnstile includes erased in the stop list for local-expand, expansion stops before analyzing the rest of the output. The point of all this erased business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.

+ +

Control returns to the begin transformer, which turns to checking/expanding the subsequent (+ almost 1), where it will encounter the identifier almost without a corresponding binding. Even though our define form produced a binding as part of its output, the expander hasn’t actually analyzed it before reaching the reference in the next expression.

+ +

The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add define to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn’t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We’re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket’s syntax model.

+ +

Internal Definition Contexts

+ +

In order for the (+ almost 1) expression from above to successfully typecheck/expand, we need to be able to associate almost with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile’s interface doesn’t suit this scenario.

+ +

Racket has the notion of an internal definition context that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.

+ +

When using local-expand, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then local-expand-ing references such as the above one should work out. Normally, Turnstile calls local-expand internally in accordance with the type rules we write down, but in order to use our own definition context we’re going to have to call it ourselves.

+ +

We can create a definition context with syntax-local-make-definition-context, as in

+ +
+ + + + +
+
+ 
1
 +
+ 
(define def-ctx (syntax-local-make-definition-context))
+
+
+
+ +

And then (imperatively) add bindings to def-ctx with syntax-local-bind-syntaxes. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier means. Passing #f corresponds to a run-time/phase 0 binding, such as that of a procedure argument, let, or define; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
 +
+ 
(define-for-syntax (int-def-ctx-bind-type-rename! x x- t ctx)
+  (syntax-local-bind-syntaxes (list x)
+                              #`(make-variable-like-transformer
+                                 (assign-type #'#,x- #'#,t #:wrap? #f))
+                              ctx)
+  (syntax-local-bind-syntaxes (list x-) #f ctx))
+
+
+
+ +

The first call binds x to a transformer that renames to x-; the second lets the expander know that we are taking care of making sure that x- will actually be bound to something.

+ +

Our define form must communicate the information needed to call int-def-ctx-bind-type-rename! back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of define that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.

+ +

Concretely, define will expand to define/intermediate, which will in turn expand to what define originally expanded to:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
 +
+ 
(define-typed-syntax (define x:id e) ≫
+  [⊢ e ≫ e- ⇒ τ]
+  #:with x- (generate-temporary #'x)
+  #:with x+ (syntax-local-identifier-as-binding #'x)
+  --------------------------------------------------
+  [⊢ (define/intermediate x+ x- τ e-) ⇒ Void])
+
+(define-syntax (define/intermediate stx)
+  (syntax-parse stx
+    [(_ x:id x-:id τ e)
+     #:with x-/τ (assign-type #'x- #'τ #:wrap? #f)
+     #'(begin-
+         (define-syntax x (make-variable-like-transformer #'x-/τ))
+         (define- x- e)
+         a-deep-dark-void)]))
+
+
+
+ +

(The reason we create an x+ using syntax-local-identifier-as-binding is due to a bug in the expander. The explanation is rather involved and frankly I only barely understand what’s going on myself (if at all), so let’s just leave it at that and move on.)

+ +

Then, for each form e in a sequence, we can call local-expand with def-ctx and then check the expansion, e-, for define/intermediate. In those cases, we can use int-def-ctx-bind-type-rename! to add it to the context. The procedure add-bindings-to-ctx! performs this check on an expanded form e- (remembering that Turnstile will wrap the output of define in an erased macro):

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
+8
 +
+ 
(define-for-syntax (add-bindings-to-ctx! e- def-ctx)
+  (syntax-parse e-
+        #:literals (erased)
+        [(erased (define/intermediate x:id x-:id τ e-))
+         (int-def-ctx-bind-type-rename! #'x #'x- #'τ def-ctx)]
+        [_
+         ;; when e expands to something other than a definition there's nothing to bind
+         (void)]))
+
+
+
+ +

We now have the key ingredients to define a procedure, walk/bind, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form e:

+ +
    +
  1. local-expand using our internal definition context, resulting in an e-.
    2. +
  2. Retrieve the type of e from the metadata of e- using Turnstile’s typeof helper.
    3. +
  3. Check if e defined a binding, in which case add it to the context.
+ +

Aggregating the expanded syntax and type of each form as we go along, we get

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
 +
+ 
(define-for-syntax (walk/bind e...)
+  (define def-ctx (syntax-local-make-definition-context))
+  (define unique (gensym 'walk/bind))
+  (define-values (rev-e-... rev-τ...)
+    (for/fold ([rev-e-... '()]
+               [rev-τ... '()])
+              ([e (in-syntax e...)])
+      (define e- (local-expand e (list unique) (list #'erased) def-ctx))
+      (define τ (typeof e-))
+      (add-bindings-to-ctx! e- def-ctx)
+      (values (cons e- rev-e-...)
+              (cons τ rev-τ...))))
+  (values (reverse rev-e-...)
+          (reverse rev-τ...)))
+
+
+
+ +

The value unique and its use as an argument is dictated by the documentation of local-expand: “For a particular internal-definition context, generate a unique value and put it into a list for context-v.” By using #'erased in the stop list for local-expand, we stop expansion at the same points that Turnstile does.

+ +

Now we can implement begin in terms of walk/bind:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
(define-typed-syntax (begin e ...+) ≫
+  #:do [(define-values (e-... τ...) (walk/bind #'(e ...)))]
+  #:with τ-final (last τ...)
+  --------------------
+  [⊢ (begin- #,@e-...) ⇒ τ-final])
+
+
+
+ +

and voilà!

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
 +
+ 
(define (add2 [x : Int])
+  (define almost (+ x 1))
+  (+ almost 1))
+
+(add2 3)
+;;=> 5
+
+
+
+ +

But Wait, There’s More

+ +

I believe this design is can be dropped in ‘as-is’ and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:

+ +
    +
  • The define form here doesn’t provide the useful shorthand for creating functions, (define (f x) e ...). Extending it to do so is relatively straightforward.
    • +
  • Supporting recursive (and mutually recursive) function definitions is a bit more complicated, but shouldn’t require many changes to the above code.
    • +
  • There’s an extensibility issue—macros that expand to multiple uses of define inside a begin won’t work (why not?), such as
+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
+8
 +
+ 
(define-syntax (define/memo stx)
+  (syntax-parse stx
+    [(_ (f [x (~datum :) τ] ...) e ...+)
+     #'(begin
+         (define memo ... memo table ...)
+         (define (f [x : τ] ...)
+           ... check memo table ...
+           e ...))]))
+
+
+
+ +

Finally, there’s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don’t have to muck around with syntax-local-bind-syntaxes and friends. If you have any ideas on this front, feel free to reach out.

+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/11/24/disappearing-code/index.html b/blog/2018/11/24/disappearing-code/index.html new file mode 100644 index 00000000..5e2d31f2 --- /dev/null +++ b/blog/2018/11/24/disappearing-code/index.html @@ -0,0 +1,217 @@ + + + + + + Disappearing Code + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Disappearing Code

+

+ :: dear diary

+

By: Ben Greenman

+
+ +

Two experiences at SPLASH 2018 reminded me that software gets thrown away and replaced.

+ + +

Story 1

+ +

The first reminder came near the end of a talk by Martin Rinard. Once upon a time, Martin was working as a consultant and a firm asked him to review a software package. (The firm wanted a second opinion about how the software computed its results.) The firm sent a zipfile; Martin found six versions of the code inside; the firm said “well, please check all six versions”; and it turned out:

+ +
    +
  • Version 1 : the source code was written in a domain-specific language (DSL) that generated code for the application
    • +
  • Version 2 : the DSL source was the same as version 1, but the generated code was slightly modified
    • +
  • …
    • +
  • Version 6 : the generated code was the source code and the DSL was gone
+ +

The moral of Martin’s story was: (1) the creators of a software system are often different from the maintainers, and (2) researchers need to build tools to help these maintainers.

+ +

Story 2

+ +

The second reminder came from a teaching assistant who said the functional programming course at their institution was currently using a Python script to test students’ code. Once upon a time, I was a teaching assistant for the same course at the same institution. We had trouble testing students’ code via the Python script left by the pre–2013 course staff, so I wrote a command-line tool to handle the tests and other compile/run/grade tasks. To keep history from repeating itself, I used the same language the course teaches (OCaml) and wrote some documentation — but it seems like that was not enough. At any rate, writing the tool was a good exercise.

+ +
+

In the end, everybody must understand for himself. — Per Martin-Löf

+ +

Reflection

+ +

In each story, the maintainers of a software system threw away some old code to make their job easier in the short term. How can we stop this “re-inventing the wheel” from happening?

+ +

Martin Rinard’s solution is to let maintenance programmers keep their current habits, but provide tools to make the short-term, pragmatic solutions into a more robust systems. Search for "failure-oblivious computing" to learn more (this was the topic of his talk).

+ +

In Story 1, the maintainers were able to avoid the DSL by modifying an inherited blob of DSL-generated code. If the DSL did not generate code, history might have taken a different course; it might be best to start with a language that offers tools for linguistic re-use, and to build a DSL from these tools — so there is no generated code. The Racket programming language is exploring this path. For a recent example, see the video-lang paper.

+ +

The Story 2 test harness, however, was not generating code. Its maintainers discarded a “big” program written in a typed functional language in favor of a script. Perhaps we need a language that allows mixing statically-typed and dynamically-typed code (shouts out to my own research).

+ +

The best solution is probably to start with a team and keep the culture alive. Always pair program!

+ +
+ +

Addendum: comment from Mitch Wand

+ +
+

The best solution is probably to start with a team and keep the culture alive. Always pair program!

+ +

Ermm, this works better for sourdough bread than for people.

+ +

Even in the not-so-real world of checking student solutions, there’s often no way of guaranteeing that one half of a pair will be around for the second round. They may be on co-op. Or the course will not be offered the next semster/year/etc. Or the course will change at the next offering (from OCaml to Python or from Racket to Java) so that large chunks of the infrastructure will have to be discarded or rewritten.

+ +

The “real” solution is to write literate code (as we preached incessantly in PDP), so that the next reader will have at least some clue as about what you wrote. This just may be sufficient incentive to modify rather than rebuild from scratch.

+ +

Ever the optimist, —Mitch

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/11/30/turnstile-mailing-list/index.html b/blog/2018/11/30/turnstile-mailing-list/index.html new file mode 100644 index 00000000..b7c29f36 --- /dev/null +++ b/blog/2018/11/30/turnstile-mailing-list/index.html @@ -0,0 +1,173 @@ + + + + + + Turnstile Mailing List + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Turnstile Mailing List

+

+ :: announcement, turnstile

+

By: Stephen Chang

+
+ +

Turnstile now has a mailing list: https://groups.google.com/forum/#!forum/turnstile-users

+ + +

There is also a #turnstile channel on the Racket Slack.

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/12/02/java-and-migratory-typing/index.html b/blog/2018/12/02/java-and-migratory-typing/index.html new file mode 100644 index 00000000..a51343da --- /dev/null +++ b/blog/2018/12/02/java-and-migratory-typing/index.html @@ -0,0 +1,456 @@ + + + + + + Java and Migratory Typing + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Java and Migratory Typing

+

+ :: migratory typing, java, transient

+

By: Ben Greenman

+
+ +

The transient approach to migratory typing (circa 2014) is similar to type erasure in Java (circa 2004) in a few interesting ways.

+ + +

Migratory typing

+ +

The goal of migratory typing is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:

+ +
    +
  • (G1) benefits from new ahead-of-time checks,
    • +
  • (G2) benefits from stronger run-time guarantees, and
    • +
  • (G3) may interact with all kinds of existing code.
+ +

There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.

+ +

A typical migratory typing system adds a static type checker to a dynamically typed language (examples), but one could also extend the type system of a statically-typed language; for example, by adding dependent types. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.

+ +

Java’s implementation of migratory typing has some interesting things in common with the transient implementation strategy recently proposed by Michael Vitousek and collaborators (DLS’14, POPL’17). The goal of this post is to demonstrate the connections.

+ +

Erasure migratory typing

+ +

Before we compare Java 1.5.0 to transient, let’s review a simpler strategy: the erasure approach to migratory typing.

+ +

TypeScript is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.

+ +
+

The DefinitelyTyped repository provides TypeScript type definitions for many JavaScript libraries.

+ +

The TypeScript compiler erases types; every type T in the source code translates to the universal “JavaScript type”. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:

+ +
(function (n0 : number, n1 : number) { return n0 + n1; })
+
+// ==(compiles to)==>
+
+(function (n0, n1) { return n0 + n1; })
+ +

TypeScript satisfies goals G1 and G3 for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal G2 because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable n0 is not guaranteed to point to a number at run-time. On one hand, this means the type annotations have no effect on the behavior of a program — and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.

+ +

In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:

+ +
    +
  • if a TypeScript expression e has the static type T and evaluates to a value v, then the only guarantee is that v is a valid JavaScript value (e.g., T could be number and v could be an incompatible object).
+ +

Transient migratory typing

+ +

Reticulated is a migratory typing system for Python that follows a transient implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal G1, the compilation to Python provides interoperability (goal G3), and the assertions partially meet goal G2.

+ +
+

These certain inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of near-constant, see On the Cost of Type-Tag Soundness, section 2.

+ +

For example, here is a Reticulated function that computes the average of a list of numbers:

+ +
# Reticulated (commit e478343)
+def average(nums : List(Float)) -> Float:
+  if ns:
+    return sum(ns) / len(ns)
+  else:
+    raise ValueError("average: expected non-empty list")
+ +

and here is the Python code it compiles to:

+ +
from retic.runtime import *
+from retic.transient import *
+from retic.typing import *
+
+def average(nums):
+    check_type_list(nums)
+    if ns:
+        return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns)))
+    else:
+        raise check_type_function(ValueError)('average: expected non-empty list')
+ +
+

Note: the Reticulated syntax for type annotations is similar to the one proposed in PEP 484, but not identical. For example, Reticulated does not require forward references to be embedded in strings.

+ +

The Reticulated compiler removes all type annotations and inserts check_type assertions throughout the code. In average, these assertions check that: (1) the input is a list, (2) the output is a float, (3) and the names sum len and ValueError point to callable values. That’s all. The assertions do not check that nums contains only floating-point numbers.

+ +
+

The assertions also do not check that the function bound to sum is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.

+ +

If nums contains something other than floating point numbers, then the call to average may cause sum to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of sum in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.

+ +

Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor C(T), e.g.:

+ +
  C(Float)                = Float
+  C(List(Float))          = List
+  C(List(Float) -> Float) = ->
+ +

Consequently, Reticulated has a slightly stronger run-time guarantee than Python:

+ +
    +
  • if e is an expression with static type T that evaluates to a value v, then v is guaranteed to have a top-level shape that matches the C(T) constructor.
+ +

Java migratory typing

+ +

Java 1.5.0 added generic types to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.

+ +

Review: generic types

+ +

Suppose we want to write a Box class that holds some kind of value; the value could be an Integer or a String or anything else. Here is a pre-generics definition:

+ +
class Box {
+  private Object val;
+
+  public Box(Object val) { this.set(val); }
+
+  public void set(Object val) { this.val = val; }
+
+  public Object get() { return this.val; }
+}
+ +

With this definition is it possible to make boxes that hold different types of values:

+ +
// good!
+Box iBox = new Box(new Integer(4));
+Box sBox = new Box(new String("X"));
+ +

but it is also possible to “change the type” of the contents of a Box:

+ +
// maybe bad!
+iBox.set(new String("not a number"));
+ +

and some calls to get must be followed by a type cast:

+ +
// annoying!
+((String) sBox.get()).charAt(0);
+ +
+ +

With generics, we can give a name (e.g. ValType) to “the type of the value inside a box”:

+ +
class GBox<ValType> {
+  private ValType val;
+
+  public GBox(ValType val) { this.set(val); }
+
+  public void set(ValType val) { this.val = val; }
+
+  public ValType get() { return this.val; }
+}
+ +

and now we can tell the type checker to check different boxes differently (satisfying goal G1):

+ +
GBox<Integer> iBox = new GBox<Integer>(new Integer(0));
+GBox<String> sBox = new GBox<String>(new String("A"));
+
+// iBox.set(new String("not a number")); // Type Error, good!
+
+sBox.get().charAt(0); // no cast, good!
+ +

Backwards compatibility & danger

+ +

Java generics are backwards-compatible with older code (goal G3). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a GBox:

+ +
// Java 1.4.0 method
+public static void evil(GBox b) { b.set(666); }
+
+// Java 1.5.0 method
+public static void test() {
+  GBox<String> sBox = new GBox<String>(new String("A"));
+  evil(sBox); // OK, but generates unchecked warning
+  sBox.get().charAt(0);
+}
+ +

The code above passes the type checker (with a warning about the evil method), and so it seems as though running the code will run the nonsense method call 666.charAt(0) and lead to evil behavior. The actual result, however, is a cast error immediately after the call sBox.get() returns.

+ +

Based on the cast error, we can tell that the compiler does not trust the type GBox<String> and inserts a run-time check that the result of the .get() is a string object.

+ +
+

“Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.” Generics in the Java Programming Language, Section 6.1

+ +

Java Type Erasure

+ +

In order to support pre-generics and post-generics code on the same virtual machine, the Java compiler erases generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the String in GBox<String> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via occurrence typing.)

+ +

After erasure, the GBox<ValType> class declaration loses its parameter:

+ +
// Erase `ValType`, replace with `Object`
+class GBox {
+  private Object val;
+
+  public GBox(Object val) { this.set(val); }
+
+  public void set(Object val) { this.val = val; }
+
+  public Object get() { return this.val; }
+}
+ +

and the client code gains a cast:

+ +
GBox sBox = new GBox(new String("A"));
+
+((String) sBox.get()).charAt(0);
+ +

So far, so good. But it’s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following “natural” definition of an ArrayList is not permitted:

+ +
class ArrayList<T> {
+  private T[] data;
+  private int size;
+
+  public ArrayList(int capacity) {
+    data = new T[capacity];
+    size = 0;
+  }
+
+  public T get(int ix) {
+    // TODO bounds check
+    return data[ix]
+  }
+
+  // ....
+}
+ +

The trouble is that T does not say anything about the data that a new array needs to handle:

+ +
ArrayList.java:6: error: generic array creation
+    data = new T[capacity];
+ +

The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:

+ +
  // possibly dangerous, if `data` is aliased to an `Object[]`
+  public ArrayList(int capacity) {
+    data = (T[]) new Object[capacity];
+    size = 0;
+  }
+ +

The other is to unsafely cast array elements in the get method, and elsewhere:

+ +
class ArrayList<T> {
+  private Object[] data;
+  private int size;
+
+  public ArrayList(int capacity) {
+    data = new Object[capacity];
+    size = 0;
+  }
+
+  public T get(int ix) {
+    boundsCheck(ix);
+    return (T) data[ix];
+  }
+
+  // ....
+}
+ +

Both may potentially lead to heap pollution.

+ +
+

"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.

+

"Ultimately, the most important motivation for this decision is compatibility with existing code." Java Language Specification, section 4.7

+ +

Run-time guarantees

+ +

By contrast to Reticulated’s C(T) transformation, the following G(T) transformation describes generic-type erasure, where T<T1> describes a type T with parameter T1 and A[T1, T2] describes a type variable A with lower bound T1 and upper bound T2:

+ +
  G(T<T1>)     = G(T)
+  G(A[T1, T2]) = G(T1)
+  G(T)         = T      otherwise
+ +

If generic-type erasure results in a type mismatch (e.g., in sBox.get().charAt(0) above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:

+ +
    +
  • if e is an expression with static type T that evaluates to a value v, then v is guaranteed to match the (bytecode) type G(T)
+ +

Discussion

+ +

TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name migratory typing describes this kind of language extension.

+ +
+

Gradual typing is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (example).

+ +

The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript libraries, however, can enforce some types.)

+ +
+

TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is MACLISP. For an erasure manifesto, see Pluggable Type Systems.

+ +

The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (POPL 2017). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The Pyret language has been successful with comparable shallow checks.

+ +
+

Note: the POPL 2017 paper advertises an “open-world soundness”, but I do not see how this idea is different from the older idea of soundness in a multi-language system (TOPLAS 2009, DLS 2006).

+ +

Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety — enough to prevent the use of a generic object from corrupting the state of a VM instance.

+ +

Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (example 1, example 2). The C# language has a similar type system and enforces generics at run-time (sources: blog post, PLDI 2001 paper, backup link to paper)

+ +

Acknowledgments

+ +

Thank you to Ryan Culpepper and Jesse Tov for noticing the similarity between Java’s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/index.html b/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/index.html new file mode 100644 index 00000000..f00488ee --- /dev/null +++ b/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/index.html @@ -0,0 +1,236 @@ + + + + + + The Behavior of Gradual Types: A User Study + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

The Behavior of Gradual Types: A User Study

+

+ :: migratory typing, gradual typing, extended abstract

+

By: Ben Greenman

+
+ + +
+

Note: this post is an extended abstract for the paper The Behavior of Gradual Types: A User Study by Preston Tunnell—Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, click here.

+ +

The long-term goal of gradual typing is to build languages that offer the “best” of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are at least three competing proposals for combining a dynamically-typed language with a similar statically-typed language.

+ +
+

It’s an interesting situation. There are dozens of papers on the semantics of gradual types—and many claim to have developers in mind—but zero papers that ask developers what they think.

+ +

To help inform the discussion, we recently designed a survey to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.

+ +

The image below shows one program from the survey:

+ +

Figure 1: example program

+ +

This program creates an array, passes it between typed and untyped variables, and performs write & read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array ("bye"). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type Array(String). A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type Number. These are the three behaviors in the survey:

+ +

Figure 2: behaviors for the example question

+ +
+

A fourth option is to reject the assignment of an Array(String) to a variable of type Array(Number). A few participants left comments asking for this behavior. See the anonymized responses for their comments, and see the paper for why we left that behavior out.

+ +

For each behavior, we asked for respondents’ preference along two independent dimensions:

+ +
    +
  • Do you like or dislike this behavior?
    • +
  • Does it match your expectation as a programmer?
+ +

Combined, the dimensions lead to four possible attitudes: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.

+ +

Figure 3: complete question

+ +

We administered the survey to three populations — software engineers, students, and Mechanical Turk workers — and thereby collected three sets of attitudes for each question. The results for the running example are below:

+ +

Figure 4: results for Question 7

+ +

The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.

+ +
+

Unlike the survey question, the behaviors in the results are labeled as Deep, Erasure, and Shallow. These names describe the three mixed-typed semantics.

+ +

For this question, the software engineers (left column, green bars) mostly picked the “Dislike and Unexpected” attitude for every behavior. The students (mid column, blue bars) also show consensus on “Dislike and Unexpected” for the Deep and Erasure behaviors; however, they are split for the Shallow behavior. The Mechanical Turk workers are divided on every behavior.

+ +

See the paper for the other questions and responses.

+ +

Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:

+ +

if you are designing a mixed-typed language and choose not to enforce full types, then make sure to explain this behavior to users!

+ +

Put lots of example programs in the language’s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.

+ +

Questions

+ +

Here are some good questions we’ve gotten that are not clearly answered in the paper.

+ +

Q. Did any respondents “expect” more than one behavior?

+ +

Yes, 59% of the software engineers and 82% of the students selected “Liked and Expected” and/or “Dislike and Expected” for different behaviors on the same program.

+ + + + + + +

Q. Did the respondents have a prior preference for static or dynamic typing?

+ +

Near the end of the survey we asked: “Which do you prefer, typed or untyped programming?”. See table 2 of the paper for coded responses to this question, or the anonymized responses for the ground truth. Most preferred typed programming.

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/01/28/on-stack-replacement/index.html b/blog/2019/01/28/on-stack-replacement/index.html new file mode 100644 index 00000000..5dcd71e6 --- /dev/null +++ b/blog/2019/01/28/on-stack-replacement/index.html @@ -0,0 +1,184 @@ + + + + + + On-Stack Replacement + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

On-Stack Replacement

+

+ ::

+

By: Ming-Ho Yee

+
+ +

Last semester, I took a course where the final project was to write a survey paper on “a topic in the intersection between computer systems and your area.” So I wrote about on-stack replacement.

+ + +

Abstract

+ +
+

On-stack replacement (OSR) is a programming language implementation technique that allows a running program to switch to a different version of code. For example, a program could start executing optimized code, and then transfer to and start executing unoptimized code. This was the original use case for OSR, to facilitate debugging of optimized code.

+

After its original use was established, OSR shifted to a different use case: optimizing programs. OSR allows the run-time system to detect if a program is executing an inefficient loop, recompile and optimize the method that contains the loop, and then transfer control to the newly compiled method. Another strategy is to optimize code based on some assumptions, then, if the assumptions are invalidated at run-time, transfer control back to the original, unoptimized code.

+

In this survey paper, we study how OSR was first introduced as a means for debugging, how it came to be used for program optimizations, its implementation as a reusable library, and other directions of research.

+ +

If you’re interested, you can find a copy here or on Overleaf.

+ +
+ +

If you liked this post, you may also be interested in tracing JITs for dynamic languages.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/02/17/writing-a-paper-with-scribble/index.html b/blog/2019/02/17/writing-a-paper-with-scribble/index.html new file mode 100644 index 00000000..7f8aed85 --- /dev/null +++ b/blog/2019/02/17/writing-a-paper-with-scribble/index.html @@ -0,0 +1,495 @@ + + + + + + Writing a paper with Scribble + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Writing a paper with Scribble

+

+ :: Scribble, tutorial

+

By: Ben Greenman

+
+ +

This post explains how to get started using Scribble to write a research paper.

+ + +
+ +
+

This post was written using Racket 7.1 and Scribble 1.29

+ +

Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:

+ +
    +
  • the paper can import definitions from the research, keeping a single point of control;
    • +
  • the language’s functional abstractions can help manage the writing;
    • +
  • the language’s drawing and/or plotting libraries can replace TikZ;
    • +
  • and you can write unit tests to validate the claims made in the paper.
+ +

Scribble, the Racket documentation tool, comes with a to-LaTeX compiler and a scribble/acmart library tailored to the new ACM paper format. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short “getting started” guide and links to some larger examples.

+ +
+

For a Scribble tutorial, see the links in: Building a Website with Scribble

+ +

Getting started with scribble/acmart

+ +

The first line of a scribble/acmart document sets the formatting options (similar to a LaTeX file using acmart.cls). For example, the GPCE 2018 call for papers asks for anonymized sigplan-format submissions with line numbers and 10 point font. The proper Scribble incantation is:

+ +
#lang scribble/acmart @sigplan @anonymous @review @10pt
+ +

Next, you may want to import some definitions. If we have a file references.rkt (see below for a definition), we can import it as follows:

+ +
@require{references.rkt}
+ +

The third main ingredient is the title and author information:

+ +
@(define neu (affiliation #:institution "Northeastern University"))
+@(define anon (email "anon@anon.net"))
+
+@title{Writing a paper with Scribble}
+@author[#:affiliation neu #:email anon]{Ben Greenman}
+
+@; optional: set the author names in the page headers
+@elem[#:style "Sshortauthors"]{B. Greenman}
+ +

The paper is now ready to be written. You can forge ahead with a new section and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and import content from other modules:

+ +
@include-abstract{abstract.scrbl}
+@include-section{introduction.scrbl}
+ +

Finally, the main page is a good place to generate the bibliography. Assuming this document imports a file like the references.rkt below, this expression inserts a bibliography titled “References”:

+ +
@generate-bibliography[#:sec-title "References"]
+ +

To build the document, invoke scribble on the command-line with the --pdf or --latex options:

+ +
$ raco scribble --pdf FILE.scrbl
+ +

If all goes well, this command generates a FILE.pdf with properly-linked cross references.

+ +

Auxiliary Files

+ +

If you save the code above to a file example.scrbl and save the files below in the same directory, then you should be able to build an example.pdf.

+ +

These files are available in a slightly different format at this link:

+ + + +

references.rkt

+ +
#lang racket/base
+
+(provide
+  ~cite citet generate-bibliography
+  fbf-icfp-2009)
+
+(require
+  scriblib/autobib)
+
+(define-cite ~cite citet generate-bibliography
+  #:style author+date-square-bracket-style)
+
+(define icfp "ICFP")
+
+(define fbf-icfp-2009
+  (make-bib
+    #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools"
+    #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler")
+    #:location (proceedings-location icfp #:pages '(109 120))
+    #:date 2017))
+ +

abstract.scrbl

+ +
#lang scribble/acmart
+
+A simple Scribble document.
+ +

introduction.scrbl

+ +
#lang scribble/acmart
+@require{references.rkt}
+
+@; start with `title` instead of `section`, because importing via
+@;  `include-section` shifts all title/section/subsections down one level
+@title{Introduction}
+
+Scribble creates a connection between a stand-alone document and the artifact
+it describes@~cite[fbf-icfp-2009].
+ +

Q. How to debug Scribble error messages?

+ +

If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.

+ +

As a compile-time example, adding @ foo to a document produces the message unexpected whitespace after @ and you can either delete the whitespace or change the @ to @"@" for a literal @-sign.

+ +

As a run-time example, adding @(+ 2 2) produces this message:

+ +
not valid in document body (need a pre-part for decode) in: 4
+ +

One fix is to convert 4 to a string, as in @~a[(+ 2 2)].

+ +

But if something goes wrong when Scribble renders a generated document to PDF, the default error output is not likely to help. For example, adding @elem[#:style "oops"] to a document produces a giant message:

+ +
$ raco scribble --pdf FILE.scrbl
+[[ ... 84K of output ... ]]
+Output written on example.pdf (1 page, 277876 bytes).
+PDF statistics:
+ 53 PDF objects out of 1000 (max. 8388607)
+ 37 compressed objects within 1 object stream
+ 7 named destinations out of 1000 (max. 500000)
+ 36877 words of extra memory for PDF output out of 42996 (max. 10000000)
+
+run-pdflatex: got error exit code
+  context...:
+  [[ ... 17 more lines ... ]]
+ +

The best way to debug these messages is to ignore them and use a LaTeX compiler directly. For the “oops” mistake, LaTeX stops at the undefined control sequence — giving a hint about how to find the problem:

+ +
$ raco scribble --latex FILE.scrbl
+$ pdflatex FILE.tex
+[[ ... 12KB of output ... ]]
+! Undefined control sequence.
+l.549 \oops
+           {}
+?
+ +

Q. How to add a LaTeX style file?

+ +

To add extra LaTeX code to the final document, create a new file and include it with the ++style command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).

+ +
$ raco scribble ++style style.tex --pdf FILE.scrbl
+ +

Here is an example style file.

+ +

style.tex

+ +
\settopmatter{printfolios=true,printccs=true,printacmref=true}
+% add page numbers etc.
+
+\overfullrule=1mm
+% draw a black rectangle near lines that overflow the margin
+ +

Another way to add extra LaTeX code is to add a tex-addition style property to the main title. This second approach makes it easy to include more than one file:

+ +
#lang scribble/acmart
+
+@require[
+  (only-in scribble/core make-style)
+  (only-in scribble/latex-properties make-tex-addition)]
+
+@(define extra-style-files
+   (list (make-tex-addition "style.tex")))
+
+@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble}
+
+@; ....
+ +

Q. How to make a figure?

+ +

Use the scriblib/figure library to add figures to a document.

+ +
@require[pict scriblib/figure]
+@figure[
+  "fig:fish"  @; figure tag, see `figure-ref`
+  @elem{A Standard Fish}  @; figure caption, appears below the content
+  @elem{fish = @(standard-fish 90 40)}]  @; content
+ +

The content of a figure can be almost anything that would work in the toplevel of the document.

+ +

Q. How to include extra files (pictures, LaTeX)?

+ +

The ++extra command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.

+ +

For example, if a document includes the content of an external LaTeX file:

+ +
@elem[#:style "input"]{inline-this.tex}
+ +

then make sure to build the document with a command like this:

+ +
$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl
+ +

inline-this.tex

+ +
% Raw LaTeX allowed here
+$\lambda x.\, x$
+ +

Q. What about in-line LaTeX?

+ +

An element with the 'exact-chars style property renders directly to LaTeX.

+ +
@(define (exact . stuff)
+   @; the style name "relax" puts a `\relax` no-op in front of the stuff
+   (make-element (make-style "relax" '(exact-chars)) stuff))
+
+@exact|{$\lambda x.\, x$}|
+@; ==> \relax{$\lambda x.\, x$}
+
+@(define ($ . math-stuff)
+   (apply exact (list "$" math-stuff "$")))
+
+@${\lambda x.\, x}
+@; ==> \relax{$\lambda x.\, x$}
+ +

Creating a #lang for a paper

+ +

For a Scribble document that is split across multiple files, it can be helpful to make a #lang that provides a common environment. Instead of starting each file with a require, e.g.:

+ +

paper.scrbl

+ +
#lang scribble/acmart
+@require["references.rkt" "helper-functions.rkt" scriblib/figure]
+
+....
+ +

files can start with a name that describes their common purpose:

+ +

paper.scrbl

+ +
#lang conference-2018-submission
+
+....
+ +

As a bonus, if the language is defined as a package then the Scribble document can use Racket’s dependency management tools:

+ +
# to install the paper and interactively install dependencies:
+$ cd conference-2018-submission;
+$ raco pkg install
+
+# To check that the paper builds with no dependency issues:
+$ raco setup --check-pkg-deps conference-2018-submission
+
+# To run all unit tests
+$ raco test -c conference-2018-submission
+ +

To create a package and language:

+ +
    +
  1. Move the Scribble document to a directory with the language name, i.e., conference-2018-submission/
    2. +
  2. Write a simple info.rkt to configure the package
    3. +
  3. Create a normal Racket module that exports the common environment
    4. +
  4. Create a conference-2018-submission/lang/reader.rkt module
+ +

Details below. For a full example, visit:

+ + + +
+ +

conference-2018-submission/info.rkt

+ +

This file defines the basic metadata for a package. For more about info.rkt, see: Tutorial: Creating a Package.

+ +
#lang info
+(define collection "conference-2018-submission")
+(define deps '("base" "scribble-lib" "at-exp-lib"))
+(define build-deps '("racket-doc" "scribble-doc"))
+(define pkg-desc "Paper for Conference 2018")
+(define version "0.1")
+ +
+ +

conference-2018-submission/main.rkt

+ +

This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the scribble/acmart language, the file “references.rkt”, and the scriblib/figure library.

+ +
#lang racket/base
+
+(provide
+  (all-from-out
+    scribble/acmart
+    scribble/acmart/lang
+    scriblib/figure
+    "references.rkt"))
+
+(require
+  scribble/acmart
+  scribble/acmart/lang
+  scriblib/figure
+  "references.rkt")
+ +
+ +

conference-2018-submission/lang/reader.rkt

+ +

This file: (1) tells Racket to use the Scribble reader on #lang conference-2018-submission modules, and (2) wraps the result of such modules in a shape that Scribble expects.

+ +
#lang s-exp scribble/base/reader
+conference-2018-submission
+#:wrapper1 (lambda (t) (cons 'doc (t)))
+ + + +

These documents use the #lang approach to writing a paper with Scribble. Check their main.rkt for example formatting functions and unit tests, and check the .scrbl files to see how the ideas above look in a larger document.

+ + + +

Finally, this repository provides a tool to start a new Scribble document:

+ + + +

Further Reading

+ + +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/index.html b/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/index.html new file mode 100644 index 00000000..15edcd0d --- /dev/null +++ b/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/index.html @@ -0,0 +1,198 @@ + + + + + + PLISS: Learn About PL Implementation in a Castle + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

PLISS: Learn About PL Implementation in a Castle

+

+ :: event, lectures, castle, language implementation

+

By: Alexi Turcotte

+
+ +

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

+ +

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

+ + +

PLISS logo

+ +

The school will feature ten speakers from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:

+ +
    +
  • Developing Security-Aware Languages with Cristina Cifuentes;
    • +
  • Semantics-First Language Design with Sylvan Clebsch;
    • +
  • Compiler Design Patterns for Machine Learning by Albert Cohen;
    • +
  • Design and Analysis of Configuration Languages by Arjun Guha;
    • +
  • A Survey of V8 and WebAssembly by Ben L. Titzer;
    • +
  • Crafting User-Friendly Compilers by Nicholas Matsakis;
    • +
  • Static Program Analysis by Anders Møller;
    • +
  • How Industry Approaches Language and Compiler Design by Joe Pamer;
    • +
  • What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.
+ +

Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.

+ +

If I may, I attended the first PLISS in 2017 and can’t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it’s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee’s testimonial here.

+ +

More information can be found at:

+ +

https://pliss2019.github.io

+ +

(No, really, it’s in a castle. Look at the pictures.)

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/04/07/forgetful-and-heedful-contracts/index.html b/blog/2019/04/07/forgetful-and-heedful-contracts/index.html new file mode 100644 index 00000000..4a78da0f --- /dev/null +++ b/blog/2019/04/07/forgetful-and-heedful-contracts/index.html @@ -0,0 +1,339 @@ + + + + + + Forgetful and Heedful contracts + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Forgetful and Heedful contracts

+

+ :: migratory typing, higher-order contracts

+

By: Ben Greenman

+
+ +

Forgetful and heedful are two methods for space-efficient contracts developed by Michael Greenberg in 2014. These methods were born in the shadow of a third method, eidetic, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.

+ + +
+ +

Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of three parties: the API code, the client code, and the contract between them.

+ +

For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:

+ +
  #lang racket/base
+
+  (module sort racket
+    (provide
+      (contract-out
+        [quicksort
+          (-> (vectorof point/c) void?)]))
+
+    (define point/c (vectorof integer?))
+
+    (define (quicksort points)
+      ....))
+
+  (module client racket
+    (require (submod ".." sort))
+    (quicksort '()))
+
+  (require 'client)
+ +
quicksort: contract violation;
+ expected a vector
+  given: '()
+  in: the 1st argument of
+      (-> (vectorof (vectorof integer?)) void?)
+  contract from: 
+      (file.rkt sort)
+  blaming: (file.rkt client)
+   (assuming the contract is correct)
+ +

That covers the basics. For an extended introduction to contracts, visit The Racket Guide.

+ +

The quicksort example and the related figures are from the paper Collapsible Contracts: Fixing a Pathology of Gradual Typing

+ +

Classic contracts and “Space Efficiency”

+ +

The (vectorof point/c) contract used above describes a possibly-mutable array whose elements match the point/c contract. Since the array can be mutated, this contract has implications for two parties:

+ +
    +
  1. the client module must supply a good array, and
    2. +
  2. the sorting module must not insert a bad element.
+ +

To enforce the second condition, the vectorof contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:

+ +
(quicksort (vector (vector 4 4)
+                   (vector 2 2)
+                   (vector 1 1)
+                   (vector 3 3)))
+ +

After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a solid blue bar for the write checks against the sort module and a striped blue bar for the read checks against the client.

+ +

A wrapped vector

+ +

In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted — because every write and read applies a contract to the element.

+ +

Layers of element wrappers

+ +

On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.

+ +

Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and “indirecting” future writes/reads through wrappers. These space and time costs can add up.

+ +
+

“on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times” — Collapsible Contracts

+ +

To fix the problem, researchers have been exploring space-efficient implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and eidetic, forgetful, and heedful are his names for three implementations.

+ +

(Although the goal of this post is to promote forgetful and heedful, we will review all three.)

+ +

Eidetic space-efficiency

+ +

The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a merge operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal “wrapping” contracts and their size is bounded by the number (and height) of source-code contracts in the program.

+ +

An eidetic contract is an N-ary tree (for N > 0):

+ +
    +
  • each node represents a higher-order contract combinator, such as vectorof
    • +
  • the N children of a node represent the different interactions that the value supports
    • +
  • each leaf is a list of non-higher-order, or flat, contracts
+ +

For example, the (vectorof point/c) source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the Collapsible Contracts paper has an illustration. Each tree node represents a vectorof contract; these nodes have N=2 children because vectors support reads and writes.

+ +

A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists — if an implementation comes with a contract-stronger? predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.

+ +

Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should contract-stronger? try to solve problems that lack polynomial-time solutions?

+ +

Thankfully, there are at least two “compromise” alternatives.

+ +

Forgetful space-efficiency

+ + + + + +
+

“Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.” — Space-Efficient Manifest Contracts

+ +

The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:

+ +
    +
  1. the client obligations from the old contract, and
    2. +
  2. the server obligations from the new contract
+ +

If not, proceed as usual — by wrapping (an unwrapped value) or raising an error. Every value receives at most one wrapper; this wrapper changes as the value flows to different clients.

+ +

Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module A exports a function f with contract (-> T1 T2) to module B, and suppose module B shares this function with a few other client modules using different contracts. As f flows to a new client, it keeps the T1 domain check and gets a replacement for the T2 codomain check.

+ +
    +
  • Keeping T1 ensures that the code inside the function (defined by module A) receives input that matches its expectation.
    • +
  • Replacing T2 ensures that each new client receives output that it expects.
+ +

Unfortunately, replacing T2 also means that clients of module B cannot trust the T2 contract. This contract is not checked, and so forgetful contracts miss some errors that would be caught by standard contracts. For the same reason, a bug in module B may go undetected by its clients — even if a later contract reports an issue, the contract system has no memory that B was partly-responsible.

+ +

Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.

+ +

Heedful space-efficiency

+ +

A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set — or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.

+ +

To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.

+ +

The heedful method is a compromise between forgetful and eidetic.

+ +
    +
  • +

    Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of contract-stronger? predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.

    • +
  • +

    Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.

+ +

For details, see the extended version of Michael’s POPL 2015 paper. Don’t bother searching the conference version — aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.

+ +

* If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to Michael Ballantyne for explaining this to me.

+ +

Priorities and Appearances

+ +

The extended version of Space-Efficient Manifest Contracts introduces the forgetful and heedful methods with extreme modesty. It’s tempting to skip past them and focus on the eidetic method.

+ +
+

“Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [Siek and Wadler 2010]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen—warm up exercises for eidetic.” — Space-Efficient Manifest Contracts

+ +

And yet, at least two other research papers rely on these “strawmen” — or rather, the ideas behind the names.

+ +

Gradual Typing with Union and Intersection Types, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.

+ +
+

“if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one” — Gradual Typing with Union and Intersection Types

+ +

This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.

+ +
+

“removing these casts preserves the soundness of the evaluation while reducing the number of them”

+

“while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits” — Gradual Typing with Union and Intersection Types

+ + + + + + +

Big Types in Little Runtime, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their transient semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.

+ +

That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value’s set. When a mismatch occurs, the semantics tries to report every type in the set that relates to the mismatch.

+ +

And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in ICFP 2018. Where will we see them next?

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/index.html b/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/index.html new file mode 100644 index 00000000..9ace8843 --- /dev/null +++ b/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/index.html @@ -0,0 +1,169 @@ + + + + + + [Conversational Concurrency (cross-post)](https://eighty-twenty.org/2018/01/24/conversational-concurrency) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Conversational Concurrency (cross-post)

+

+ :: dissertation

+

By: Tony Garnock-Jones

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/09/05/lexical-and-dynamic-scope/index.html b/blog/2019/09/05/lexical-and-dynamic-scope/index.html new file mode 100644 index 00000000..7da90526 --- /dev/null +++ b/blog/2019/09/05/lexical-and-dynamic-scope/index.html @@ -0,0 +1,319 @@ + + + + + + Lexical and Dynamic Scope + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Lexical and Dynamic Scope

+

+ :: scope, definitions, history

+

By: Ming-Ho Yee

+
+ +

This all started with a simple question about the R programming language: is R lexically or dynamically scoped?

+ +

To answer that question, we need to understand what scope is, along with lexical scope and dynamic scope.

+ + +

In this blog post, I’d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a subsequent post, I’ll discuss scoping in R and why it can be confusing.

+ +

What is scope?

+ +

Scope refers to the places in a program where a variable is visible and can be referenced.

+ +

An interesting situation is when a function has free variables. Consider the example below:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
 +
+ 
x <- 1
+f <- function(a) x + a
+g <- function() {
+  x <- 2
+  f(0)
+}
+g() # what does this return?
+
+
+
+ +

On line 1, we create a mapping for x with value 1. On line 2, we define a function f whose body uses the parameter a, but also the free variable x. On line 3, we define a function g, whose body creates a new mapping for x with value 2, and then calls f(0). (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call g().

+ +

What value does g return when it is called? What mapping does the free variable x on line 2 refer to? Does it refer to the mapping on line 1 that was visible when f was defined? Or does it refer to the mapping on line 4 that was created just before f was called?

+ +

Lexical scoping

+ +

Under lexical scoping (also known as static scoping), the scope of a variable is determined by the lexical (i.e., textual) structure of a program.

+ +

In the example above, the definition of x on line 1 creates a scope that starts after its definition and extends into the bodies of f and g. However, the second definition of x on line 4 creates a new scope that (1) shadows the previous definition of x, and (2) does not extend into the call f(0) on line 5. Looking at this from another direction, the use of x on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.

+ +

Therefore, under lexical scoping, the example program returns 1.

+ +

Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which definition each variable refers to—but it may not be able to determine the values of each variable.

+ +

Dynamic scoping

+ +

Under dynamic scoping, a variable is bound to the most recent value assigned to that variable, i.e., the most recent assignment during the program’s execution.

+ +

In the example above, the free variable x in the body of f is evaluated when f(0) is called on line 5. At that point (during program execution), the most recent assignment was on line 4.

+ +

Therefore, under dynamic scoping, the example program returns 2.

+ +

Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.

+ +

(Edited 2020/08/13: As of Emacs 27.1, “lexical binding is now used by default when evaluating interactive Elisp.” Thanks to Artem Pelenitsyn for bringing this to my attention.)

+ +

Now for a digression

+ +

These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.

+ +

However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with—a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.1

+ +

I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for dynamic scope not mention scope at all? If scope is about “where a variable is visible,” and that definition is with respect to a variable definition, then why do so many explanations and examples define lexical and dynamic scope in terms of variable use?

+ +

Scope and Extent

+ +

I found some answers in Guy Steele’s Common Lisp the Language, 2nd Edition,2 which Matthias Felleisen recommended to me.

+ +

In chapter 3, Steele introduces the concepts of scope and extent:

+ +
+

Scope refers to the spatial or textual region of the program within which references may occur. Extent refers to the interval of time during which references may occur.

+ +

In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:

+ +
    +
  • +

    Lexical scope: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, e.g., the body of a variable definition.

    • +
  • +

    Indefinite scope: a reference can occur anywhere in the program.

    • +
  • +

    Dynamic extent: a reference can occur during the time between an entity’s creation and its explicit destruction, e.g., when a local variable is created upon entering a function and destroyed when returning from that function.

    • +
  • +

    Indefinite extent: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)

+ +

Steele points out that dynamic scope is a misnomer, even though it is both a traditional and useful concept. It can be defined as indefinite scope and dynamic extent. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.

+ +

Discussion

+ +

I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of “dynamic scope”—it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!

+ +

The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.

+ +

A brief and incomplete history of lexical scope

+ +

During my research of different definitions of lexical scope, I began to wonder if there was an “original” definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.

+ +
    +
  • +

    1960. John McCarthy publishes the original paper on Lisp.3 In History of Lisp,4 McCarthy writes that he borrowed the λ-notation from Alonzo Church’s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the “FUNARG device.”

    • +
  • +

    1963. After a few years of work, the Revised Report on Algorithm Language ALGOL 60 is published.5 While “lexical scope” is not explicitly mentioned, it is recognizable in the specification.

    • +
  • +

    1964. Peter Landin shows how expressions in programming languages can be modelled in Church’s λ-notation.6 He also introduces the concept of a closure, which pairs a lambda expression with the environment it was evaluated in.

    • +
  • +

    1970. Joel Moses describes the problem of free variables in functions.7 He considers both the “downward” case (where a function is passed to another function) and the “upward” case (where a function returns a function), and remarks on the correspondence between Lisp’s FUNARG device and Landin’s closures.

    • +
  • +

    1975. Gerald Sussman and Guy Steele publish the first Scheme paper.8 They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL’s.

    • +
  • +

    1978. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.9 In particular, they provide a detailed discussion on lexical and dynamic scoping.

+ +

Next stop, R

+ +

Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. Is R lexically or dynamically scoped?

+ +

In the next blog post, we’ll answer that question, and also see how R can be very confusing.

+ +

I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.

+ +
+ +
+
    +
  1. +

    For example, at one point I defined lexical/dynamic scoping in terms of a “lexical environment” and a “dynamic environment.” But (1) that’s a circular definition, (2) it assumes the reader has some intuition of how a “lexical environment” is different from a “dynamic environment,” and (3) it conflates two different kinds of “environment.” ↩

    2. +
  2. +

    G. Steele. “Scope and Extent,” in Common Lisp the Language, 2nd ed. 1990. [Available online] ↩

    3. +
  3. +

    J. McCarthy. “Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,” Communications of the ACM, vol. 3, no. 4, April 1960. [DOI][Available online] ↩

    4. +
  4. +

    J. McCarthy. “History of LISP,” in History of Programming Languages, 1978. [DOI][Available online] ↩

    5. +
  5. +

    P. Naur (ed.). “Revised Report on Algorithmic Language ALGOL 60,” Communications of the ACM, vol. 6, no. 1, January 1963. [DOI][Available online] ↩

    6. +
  6. +

    P. Landin. “The mechanical evaluation of expressions,” The Computer Journal, vol. 6, no. 4, January 1964. [DOI][Available online] ↩

    7. +
  7. +

    J. Moses. “The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,” SIGSAM Bulletin 15, July 1970. [DOI][Available online] ↩

    8. +
  8. +

    G. Sussman and G. Steele. “SCHEME: An Interpreter for Extended Lambda Calculus.” 1975. [Available online] ↩

    9. +
  9. +

    G. Steele and G. Sussman. “The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).” 1978. [Available online] ↩

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/09/10/four-kinds-of-scoping-in-r/index.html b/blog/2019/09/10/four-kinds-of-scoping-in-r/index.html new file mode 100644 index 00000000..3c644f7b --- /dev/null +++ b/blog/2019/09/10/four-kinds-of-scoping-in-r/index.html @@ -0,0 +1,488 @@ + + + + + + Four Kinds of Scoping in R + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Four Kinds of Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the first and second parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.

+ +

In this third and final part of my blog series, I’d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.

+ +

This is a “bonus” blog post, because I’m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.

+ + +

Lexical scope and statistical computation

+ +

In Lexical Scope and Statistical Computation,1 Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.

+ +

For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:

+ +
    +
  • trivial: no free variables allowed
    • +
  • static: a free variable takes its value from a set of global variables
    • +
  • lexical: a free variable takes the value of the binding that was in effect when the function was defined
    • +
  • dynamic: a free variable takes the value of the most recent assignment to that variable
+ +

Note that under this set of definitions, static scoping is a separate scoping rule and not another name for lexical scoping.

+ +

It is possible to simulate each of strategies in R. For fun, we can even construct “factories” that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)

+ +

Template

+ +

Our examples will follow the template given below:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
 +
+ 
factory <- function(fun) {
+  <???>
+}
+
+x <- 0
+h <- function() {
+  x <- 1
+  factory(function(a) x+a)
+}
+g <- h()
+f <- function() {
+  x <- 2
+  g(0)
+}
+f() # error, 0, 1, or 2
+
+
+
+ +

We want to define a factory that takes a function literal and returns a closure that implements the desired scoping rule.

+ +

Our example consists of three definitions of x. On line 5, we assign 0 to x at the top level. On line 7, we assign 1 to x inside function h, where we also create the closure. On line 12, we assign 2 to x inside the function f and right before we call g, which is the closure.

+ +

Finally, we call f and observe the result:

+ +
    +
  • Under trivial scoping, no free variables are allowed, so f() should result in an error.
    • +
  • Under static scoping, free variables may only refer to global variables, so f() should return 0.
    • +
  • Under lexical scoping, free variables refer to the variables in scope when the function was defined, so f() should return 1.
    • +
  • Under dynamic scoping, free variables take the value from the most recent assignment, so f() should return 2.
+ +

We will implement the body of factory in only 3–5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to factory on line 10.

+ +

Trivial scoping

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
 +
+ 
makeTrivial <- function(fun) {
+  res <- eval(substitute(fun))
+  environment(res) <- baseenv()
+  res
+}
+
+x <- 0
+h <- function() {
+  x <- 1
+  makeTrivial(function(a) x+a)
+}
+g <- h()
+f <- function() {
+  x <- 2
+  g(0)
+}
+f() # Error in f(0) : object 'x' not found
+
+
+
+ +

substitute returns the unevaluated parse tree for fun. In other words, it obtains the literal argument that was passed for fun. This works because of call-by-need semantics in R: function arguments are packaged up into promises. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek2 discusses laziness in R in more detail.

+ +

In this example, on line 8, we call factory with function(a) x+a as the argument for the formal parameter fun. Then, we evaluate that parse tree with eval.

+ +

At this point, res is the closure with expression function(a) x+a and a reference to the environment of makeTrivial. On line 3, we change that reference to baseenv(), which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.

+ +

Therefore, variable lookup in the function literal will only search the base environment, so f() results in an error.

+ +

Static scoping

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
 +
+ 
makeStatic <- function(fun) {
+  res <- eval(substitute(fun))
+  environment(res) <- globalenv()
+  res
+}
+
+x <- 0
+h <- function() {
+  x <- 1
+  makeStatic(function(a) x+a)
+}
+g <- h()
+f <- function() {
+  x <- 2
+  g(0)
+}
+f() # 0
+
+
+
+ +

For this example, on line 3, we update the environment of res to refer to globalenv(), which is the top-level environment where globals are defined.

+ +

Therefore, variable lookup searches the top-level environment, so f() returns 0.

+ +

Lexical scoping

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
 +
+ 
makeLexical <- function(fun) {
+  res <- eval(substitute(fun))
+  environment(res) <- parent.frame()
+  res
+}
+
+x <- 0
+h <- function() {
+  x <- 1
+  makeLexical(function(a) x+a)
+}
+g <- h()
+f <- function() {
+  x <- 2
+  g(0)
+}
+f() # 1
+
+
+
+ +

Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of res to parent.frame(), which is the environment of the function (h) that called the current function (makeLexical). This allows us to simulate lexical scoping, as if the function literal was evaluated inside h, rather than makeLexical.

+ +

Therefore, variable lookup searches the environment of h, so f() returns 1.

+ +

Dynamic scoping

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
 +
+ 
makeDynamic <- function(fun) {
+  function(...) {
+    res <- eval(substitute(fun))
+    environment(res) <- parent.frame()
+    res(...)
+  }
+}
+
+x <- 0
+h <- function() {
+  x <- 1
+  makeDynamic(function(a) x+a)
+}
+g <- h()
+f <- function() {
+  x <- 2
+  g(0)
+}
+f() # 2
+
+
+
+ +

For this example, we need another level of indirection. makeDynamic returns an anonymous function literal. The anonymous function takes ..., which represents an arbitrary list of arguments, and then on line 5 we call res with those exact arguments. Note that we set the environment of res to be the environment of the caller of the anonymous function. Because of the multiple levels of indirection, the caller is f, on line 17.

+ +

On line 12, makeDynamic returns a closure for the anonymous function. h returns that closure when it is called, and assigns it to g. When g is called on line 17, the function literal function(a) x+a is finally evaluated, and its environment is set to the environment of f, the caller of g.

+ +

Therefore, variable lookup searches the environment of f, so f() returns 2.

+ +

Conclusion

+ +

Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the previous post, it’s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.

+ +

And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.

+ +

I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.

+ +
+ +

References

+ +
+
    +
  1. +

    R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, Journal of Computational and Graphical Statistics, vol. 9, no. 3, 2000. [DOI][Available online] ↩

    2. +
  2. +

    A. Goel and J. Vitek. “On the Design, Implementation and Use of Laziness in R,” in Proceedings of the ACM in Programming Languages (PACMPL), vol. 3, no. OOPSLA, 2019. To appear. [Available online] ↩

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/09/10/scoping-in-r/index.html b/blog/2019/09/10/scoping-in-r/index.html new file mode 100644 index 00000000..1917c729 --- /dev/null +++ b/blog/2019/09/10/scoping-in-r/index.html @@ -0,0 +1,674 @@ + + + + + + Scoping in R + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the previous post of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: is R lexically or dynamically scoped?

+ + +

Recall the example program from before:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
 +
+ 
x <- 1
+f <- function() x
+g <- function() {
+  x <- 2
+  f()
+}
+g() # what does this return?
+
+
+
+ +

Let’s examine what happens when we run this example. First, we create a mapping for x in the top-level environment. On line 2, we define a function f, which returns the value of some x. On line 3, we define a function g, which creates a new mapping for x, and then calls f. Note that the assignment on line 4 does not update the definition on line 1.

+ +

When f is called, it needs to look up the value of x. In other words, does the reference of x on line 2 refer to the assignment on line 1 or the assignment on line 4? If f returns 1, then the behaviour matches lexical scoping. If it returns 2, then the behaviour matches dynamic scoping.

+ +

When we run this example, the result is 1. This implies that R is lexically scoped.

+ +

But there’s more to this story. In the rest of this blog post, I’ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.

+ +

The next and final part of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.

+ +

R is lexically scoped, but…

+ +

In Evaluating the Design of the R Language,1 Morandat, Hill, Osvald, and Vitek write:

+ +
+

As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.

+ +

In other words, R provides many different “escape hatches”—ways to bypass lexical scoping. Additionally, even without escape hatches, some of R’s functionality can be surprising.

+ +

Functions, environments, and variables in R

+ +

Before we look at some examples, I think it’s useful to briefly discuss some of the core concepts in R that relate to scoping.

+ +
    +
  • +

    Functions. R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.

    • +
  • +

    Environments. An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the “enclosing” environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.

    • +
  • +

    Variable lookup. When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.

    • +
  • +

    Variable assignment. <- is the variable assignment operator in R. The expression x <- 1 assigns the value 1 to the variable x in the current environment. If a mapping for x already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.

+ +

From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.2

+ +

On the other hand, variable lookup depends on the run-time state of the program—names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.

+ +

The following examples will make this clear.

+ +

Examples

+ +

Adding variable mappings at run time

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
 +
+ 
x <- 1
+f <- function() {
+  g <- function() x
+  x <- 2
+  g()
+}
+f() # 2
+
+
+
+ +

When f is called, it creates a function g that returns x, assigns 2 to x, and then calls g. When g is called, it looks up x. Since no mapping is found in g’s environment, it searches in the enclosing environment (f’s), and finds that x has value 2. Therefore, g returns 2.

+ +

Note that the x on line 3 is resolved only when function g is called, not when it is defined. However, when g is defined, its environment has a reference to f’s environment. Therefore, as long as x is defined before g is called, the lookup will always succeed.

+ +

Here’s a second example:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
+8
 +
+ 
x <- 1
+f <- function(b) {
+  if (b)
+    x <- 2
+  x
+}
+f(TRUE)  # 2
+f(FALSE) # 1
+
+
+
+ +

f is a function that branches on its argument, b. If b evaluates to true, then the expression x <- 2 is evaluated, and a mapping for x is created in f’s environment. Otherwise, no mapping is created.

+ +

When we look up the value of x on line 5, R will first search the function’s environment. If b evaluated to true, then R will find a value for x, which is 2. Otherwise, R will search in the enclosing environment of f, and find that x is 1.

+ +

Both of these examples vaguely resemble dynamic scoping, in that x takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.

+ +

Function lookup

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
f <- function(x) x
+g <- function(f) {
+  f(0) # not an error
+}
+g(42) # 0
+
+
+
+ +

R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.

+ +

In this example, we call g with the argument 42, which is not a function. Then, in the body of g, we call f(0) on line 3, which requires looking up f. Although there is an f in the environment of g, its value is 42, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so f(0) returns 0.

+ +

This behaviour exists because c is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.

+ +

Super assignment

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
+8
 +
+ 
x <- 0
+f <- function() {
+  x <- 1
+  x <<- 2
+  x
+}
+f() # 1
+x   # 2
+
+
+
+ +

<<- is the “super assignment” operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.

+ +

In the above program, we define x to be 0 at the top level, and then define the function f. When we call f on line 7, it assigns 1 to x on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, f returns x, which is looked up from the local environment and has value 1. Finally, line 8 looks up x from the top level environment, which has value 2.

+ +

Evaluating arbitrary code

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
 +
+ 
x <- 1
+f <- function(t) {
+  eval(parse(text = t))
+  x
+}
+g <- function() {
+  x <- 2
+  f("x <- 0")
+}
+g() # 0
+
+
+
+ +

R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument t, which happens to be the string "x <- 0". Then, when line 4 executes, the lookup of x returns 0.

+ +

Simulating dynamic scope

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
+8
+9
 +
+ 
x <- 1
+f <- function() {
+  get("x", envir = parent.frame())
+}
+g <- function() {
+  x <- 2
+  f()
+}
+g() # 2
+
+
+
+ +

On line 3, we perform an explicit variable lookup for x, but we do so in the environment parent.frame(), which refers to the calling function’s environment, in this case, g’s environment.. Therefore, the lookup returns 2.

+ +

Note that R has a similarly named function, parent.env(e) which returns the enclosing environment of the given environment e.

+ +

Constructing an arbitrary environment

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
 +
+ 
x <- 1
+f <- function() {
+  e <- new.env()
+  e$x <- 3
+  get("x", envir = e)
+}
+g <- function() {
+  x <- 2
+  f()
+}
+g() # 3
+
+
+
+ +

When f is called, it constructs a new environment, e, which is initially empty. (By default, its enclosing environment is the current environment, which is f’s.) Next, on line 4, it directly adds a mapping to that environment, assigning 3 to x. Then, on line 5, the lookup is explicitly done in environment e, so f returns 3.

+ +

Deleting mappings

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
 +
+ 
x <- 1
+f <- function() {
+  rm("x", envir = parent.env(environment()))
+  x
+}
+g <- function() {
+  x <- 2
+  f()
+}
+g() # Error in f() : object 'x' not found
+
+
+
+ +

Not only is it possible to dynamically add and modify mappings in R, but it is also possible to delete mappings. This is what line 3 does: it explicitly removes the mapping for x from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when f is called, the lookup of x fails and an error is raised.

+ +

Infinite loop during variable lookup

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
+6
+7
 +
+ 
enva <- new.env()
+envb <- new.env()
+parent.env(enva) <- envb
+parent.env(envb) <- enva
+f <- function() x
+environment(f) <- enva
+f()
+
+
+
+ +

In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.

+ +

On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: enva’s enclosing environment is envb, and envb’s enclosing environment is enva.

+ +

On line 5, we define a function with a free variable, x, but on line 6, we set f’s environment to be enva. Finally, we call f.

+ +

When the body of f is evaluated, it needs to look up x. Lookup starts in f’s environment, which we set to be enva. Since no mapping for x is found, lookup continues in enva’s enclosing environment, which is envb. However, envb is also empty, so lookup continues in its enclosing environment, which is enva, and now lookup results in an infinite loop.

+ +

An intuition for scoping in R

+ +

Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
 +
+ 
# example 1
+x <- 1
+f <- function() {
+  g <- function() x
+  x <- 2
+  g()
+}
+f() # 2
+
+# example 2
+x <- 1
+f <- function(b) {
+  if (b)
+    x <- 2
+  x
+}
+f(TRUE)  # 2
+f(FALSE) # 1
+
+
+
+ +

It seems that x takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What’s missing from our intuition?

+ +

The key insight is that R is function scoped. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <- does not create a new scope, and it is more useful to think of it as a mutation on the current environment. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)

+ +

In a sense, it might be more accurate to say that R environments are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.

+ +

Discussion

+ +

All of this might make you a little uncomfortable, and uncertain about R’s scoping rules.

+ +

On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply “lexical-like” behaviour, and the creators have confirmed that lexical scoping was the intent.

+ +

On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment—but this is not consistent with R’s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.

+ +

This ambiguity shows up in a paper,3 where the authors write:

+ +
+

Furthermore, because variable scoping in R is dynamic and can be modified at the language level […] it cannot be trivially guaranteed that x is going to point to the same data structure throughout the entire execution of the loop.

+ +

It is true that a variable x may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.

+ +

It is true that variable lookup is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of dynamic scope, then it would be fair to say that R is dynamically scoped.

+ +

But if your definition of dynamic scope is “a variable is bound to the most recent assignment during the program’s execution,” then it is not correct to say R is dynamically scoped.

+ +

I think we have this ambiguity because scope (the places in a program where a variable can be referenced) and variable lookup or name resolution (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.

+ +

Personally, I prefer the definition of scope that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is both lexically and dynamically scoped, or that R is neither lexically and dynamically scoped.

+ +

I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language—when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.

+ +

Regardless, it is very easy to miscommunicate, so I think it’s important to be very clear and make sure you and your audience know what definitions of scoping you’re using!

+ +

Conclusion

+ +

This entire adventure started when we were working on a paper,4 and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!

+ +

This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.

+ +

To summarize, I would say that scope refers to the places in a program where a variable is visible and can be referenced. Under lexical scoping, the scope of a variable is determined by the lexical (i.e., textual) structure of a program. Under dynamic scoping, a variable is bound to the most recent value assigned to that variable, i.e., the most recent assignment during the program’s execution.

+ +

I would say that R aims to be lexically scoped—it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.

+ +

Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!

+ +

If you want more content about R and scoping, the third and final part of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.

+ +

Edited 2020/02/21: For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), this blog post has some nice examples and diagrams.

+ +

I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.

+ +

If you liked this post, you may also be interested in the following Twitter threads about R: one, two and three.

+ +
+ +

References

+ +
+
    +
  1. +

    F. Morandat, B. Hill, L. Osvald, J. Vitek. “Evaluating the Design of the R Language,” in Proceedings of the European Conference on Object-Oriented Programming (ECOOP), 2012. [DOI][Available online] ↩

    2. +
  2. +

    R. Gentleman and R. Ihaka. “Lexical Scope and Statistical Computing”, Journal of Computational and Graphical Statistics, vol. 9, no. 3, 2000. [DOI][Available online] ↩

    3. +
  3. +

    L. Stadler, A. Welc, C. Humer, and M. Jordan. “Optimizing R Language Execution via Aggressive Speculation,” in Proceedings of the Symposium on Dynamic Languages (DLS), 2016. [DOI] ↩

    4. +
  4. +

    O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. “R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,” in Proceedings of the Symposium on Dynamic Languages (DLS), 2019. To appear. [Available online] ↩

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/10/31/complete-monitors-for-gradual-types/index.html b/blog/2019/10/31/complete-monitors-for-gradual-types/index.html new file mode 100644 index 00000000..21f745e1 --- /dev/null +++ b/blog/2019/10/31/complete-monitors-for-gradual-types/index.html @@ -0,0 +1,258 @@ + + + + + + Complete Monitors for Gradual Types + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Complete Monitors for Gradual Types

+

+ :: migratory typing, gradual typing, complete monitoring, extended abstract

+

By: Ben Greenman

+
+ +

Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction — a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.

+ + +
+

Note: this post is an extended abstract for the paper Complete Monitors for Gradual Types by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, click here.

+ +

Example: Clickable Plot

+ +

The program below has a subtle bug. Can you find it?

+ +

Untyped client code, a typed API, and untyped library code.

+ +

First of all, this pseudocode program combines three chunks of code:

+ +
    +
  • +

    On the left, an untyped client script defines a function h that expects a pair of numbers and returns an image. The client uses this function to create a ClickPlot object, and then displays the plot — ideally in a new GUI window.

    • +
  • +

    In the center, a typed API file describes a ClickPlot object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The mouseHandler method expects a MouseEvt object and returns nothing. The show method expects no arguments and returns nothing. (Presumably, these methods have side effects.)

    • +
  • +

    On the right, an untyped library module implements a ClickPlot object. Most of the code is omitted (...), but the mouseHandler method sends its input directly to the onClick callback.

+ +

The bug is in the API — in the type ([N, N]) => Image. This type promises that a given function can expect a pair of numbers, and indeed the client function h expects a pair. But the library code on the right sends a MouseEvt object.

+ +

What happens when we run this program in a type-sound mixed-typed language? Does h receive the invalid input?

+ +

As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.

+ +

Type Soundness is Not Enough

+ +

Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.

+ +
+

Classic Type Soundness

+

If e : T then one of the following holds:

+
    +
  • e -->* v and v : T
    • +
  • e diverges
    • +
  • e -->* OkError
+ +

A mixed-typed language needs two “type soundness” theorems: one for typed code and one for untyped code. The typed soundness theorem can resemble a classic theorem. The untyped soundness theorem is necessarily a weaker statement due to the lack of types:

+ +
+

Mixed-Typed Soundness

+

If e : T then one of the following holds:

+
    +
  • e -->* v and v : T
    • +
  • e diverges
    • +
  • e -->* OkError
+

And if e is untyped then one of the following holds:

+
    +
  • e -->* v and v is an untyped value
    • +
  • e diverges
    • +
  • e -->* OkError
+ +

Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback h in the code above receives a pair value. We have an untyped function called from an untyped context — since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!

+ +

Untyped library sends input directly to untyped client.

+ +

Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type [N, N] to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.

+ +

The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.

+ +

Complete Monitoring

+ +

A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of ownership to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.

+ +
+

Complete Monitoring

+

For all well-formed e and all e', if e -->* e' then every subexpression of e' has a unique owner.

+ +

This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.

+ +

An Aid to Debugging

+ +

The question raised by the Clickable Plot example is whether a language can detect one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging — either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.

+ +

The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.

+ +

Also in the paper, you can find:

+ +
    +
  • a model of ownership, clear laws for how ownership changes during evaluation;
    • +
  • examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;
    • +
  • definitions for blame soundness and blame completeness;
    • +
  • an analysis of three semantics, which correspond to Typed Racket, Transient Reticulated, and a compromise;
    • +
  • and discussion of an alternative, heap-based model of ownership.
+ +

Paper: https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2019/12/12/prl-offsite-2019-retrospective/index.html b/blog/2019/12/12/prl-offsite-2019-retrospective/index.html new file mode 100644 index 00000000..ae6c6f09 --- /dev/null +++ b/blog/2019/12/12/prl-offsite-2019-retrospective/index.html @@ -0,0 +1,284 @@ + + + + + + PRL Offsite 2019 Retrospective + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

PRL Offsite 2019 Retrospective

+

+ :: offsite

+

By: Ben Greenman, Olek Gierczak

+
+ +

On November 11th 2019, the PRL had a private offsite meeting at the More Than Words bookstore in downtown Boston. For future offsite organizers, this post records what happened and how.

+ + +

Early Planning, Goals

+ +

Every Fall, the PRL holds a kickoff meeting to assign roles for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.

+ +
+

Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the Warren Conference Center is Northeastern University’s only franchised off-campus venue.

+ +

The motivation for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students — both to get to know the group, and to become known. They’re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the Spring 2017 edition of HOPL.

+ +

Our primary goal this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.

+ +

In light of the main goal, we prefer to call the event an “offsite” (or “offsite meeting”) rather than a “retreat” because the target was an informative day rather than a relaxing one.

+ +

Booking and Logistics

+ +

Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that’s all we had.

+ +

Date = Nov 11

+ +

In the beginning, we chose Veterans’ Day (2019–11–11) and Northeastern reading day (2019–12–05) as possible dates. We ended up with Veterans’ Day.

+ +

A small number of lab members were opposed to Veterans’ Day. They gave two reasons: the Fall semester is especially busy, and Veterans’ Day is a federal holiday.

+ +

Venue = MTW

+ +

The first venue we tried was the Warren Conference Center in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the Khoury Events Team to reach out on our behalf. They returned with an email address and event menu (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.

+ + +

Second, we contacted the Northeastern University Alumni Center. They were not open on Veterans’ Day 2019.

+ +

Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: More Than Words.

+ +

More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!

+ +
+

More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you’ll see a mix of “employees” and “volunteers” helping out.

+ +

Booking was complicated, though, by the fact that Northeastern requires liability insurance for all off-campus events. If you are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.

+ +

Here are more facts about MTW:

+ +
    +
  • the theater space seats 30 with lots of extra space
    • +
  • the two long tables in the reading room can seat about 20
    • +
  • the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor
    • +
  • there’s no whiteboard, but MTW can supply an easel stand
    • +
  • much of the furniture in store is antique and for sale, so be careful using it — both to avoid damaging it, and because antiques can be oddly-shaped
    • +
  • the bookstore was closed on Veterans’ Day, but we were able to have our event and buy books
    • +
  • MTW may have fridge space to temporarily hold leftovers
    • +
  • closest T station: Tufts Medical Center
+ +

Keynote = None

+ +

The original, ambitious plan was to invite two keynote speakers — one working in industry and one from academia — to enrich the offsite with new knowledge. And because this was the PRL’s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)

+ +

We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.

+ +

Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).

+ + +

Food, Coffee, Tea

+ +

Flour Bakery + Cafe provided lunch. For most days, you can order from Flour using an online form. For holidays, you may need to send an email — that’s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan & gluten-free everything spiced salads.

+ +

Cuppacoffee provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything — the order + napkins, milk, cream cheese, butter, and knives — ourselves because Cuppacoffee is very close to MTW.

+ +

CMart sold us water and juices. (There is also a Whole Foods near MTW.)

+ +

At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour’s sticky buns.

+ +

Event Program

+ +

The following assumptions/goals constrained the schedule for the day:

+ +
    +
  • give all 17 on-campus Ph.D. students a talk slot; talks must either communicate one technical idea from recent work or say what led the speaker to apply to a PL Ph.D. program
    • +
  • allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)
    • +
  • do not make the audience sit for more than 1 hour at a time
    • +
  • maximize the number and length of breaks (to encourage discussions)
    • +
  • include a relevant and fun welcome activity
    • +
  • save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions
+ +

We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.

+ +

The activity was codewalks. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the IntegerDivider class from a silly FizzBuzz implementation and a solution to the Dutch national flag problem). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and egoless programming.

+ +

The talks ran conference-style. One organizer played “session chair” to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.

+ +

For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.

+ +

Looking Back at the Details

+ +

Reading Day in the Spring may be a good, free date for future retreats.

+ +

We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.

+ +

The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we’d keep the morning breaks short — just enough to use the restroom and grab a coffee — and aim for 30-min breaks in the afternoon.

+ +

Groups of three 15-minute talks worked well, but groups of four talks might be equally good.

+ +

Perhaps each speaker should get the chance to pick a talk length. NEPLS, for example, allows a choice between 5-min. and 30-min. talks.

+ +

The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick “topics” beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.

+ + +

Looking Back, Overall

+ +

This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.

+ +

There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time — keep the same date & program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.

+ +

With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/index.html b/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/index.html new file mode 100644 index 00000000..561fb6dd --- /dev/null +++ b/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/index.html @@ -0,0 +1,577 @@ + + + + + + The Typed Racket Optimizer vs. Transient + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

The Typed Racket Optimizer vs. Transient

+

+ :: typed racket, transient, offsite

+

By: Ben Greenman

+
+ +

What type-directed optimizations does Typed Racket perform and do any require full types?

+ + +
+

This post is based on a short talk. Slides from the talk are here: http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf

+ +

Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [JFP 2019]. And so, the standard types are very strong but (possibly) slow.

+ +

Lately, I’ve been working on a transient back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee — only that typed code cannot get “stuck” — via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [ICFP 2018], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.

+ +

The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:

+ +
    +
  1. a static type checker,
    2. +
  2. a compiler from types to contracts, and
    3. +
  3. a type-driven optimizer [PADL 2012, OOPSLA 2012].
+ +

Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?

+ +

Q. Can Transient re-use the Typed Racket optimizer?

+ +

The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.

+ +
    +
  • In Standard Typed Racket, if an expression e has type T and reduces to a value v (for short, e : T —>* v), then the result v definitely matches the full type T.
    • +
  • In Transient Typed Racket, if e : T —>* v then the result v matches the toplevel “shape” of T but (maybe) nothing more.
+ +

The idea of a “shape” is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.

+ +

Here are a few shapes according to the current Transient prototype:

+ +
  Shape(Natural)                = Natural
+  Shape(Listof String)          = Listof Any
+  Shape(Symbol -> Boolean)      = Any -> Any
+  Shape(Vector Void Void)       = Vector Any Any
+  Shape(U Void (Listof Symbol)) = U Void (Listof Any)
+ +

For the current shapes, can we re-use the Typed Racket optimizer?

+ +

Optimization Topics

+ +

Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of “safe” or “unsafe” for Transient.

+ +

To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.

+ +
+ +

Topic 1: apply

+ +

apply.rkt “inlines” expressions of the form (apply f (map g xs)) to map and fold in one pass over the list (xs). Currently, the pass only triggers when f is + or *.

+ +

Example

+ +
  ;; Type Assumptions
+  (: xs (Listof Integer))
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (apply + (map abs xs))
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (let loop ((v 0)
+             (lst xs))
+    (if (null? lst)
+      v
+      (loop (+ v (abs (unsafe-car lst)))
+            (unsafe-cdr lst))))
+ +

Verdict: safe, but risky.

+ +

Technically, this transformation is unsound for Transient because of how it uses unsafe-car. The expansion of (apply * (map g xs)) applies (g (unsafe-car xs)) without confirming that the first element of xs matches its expected type. This unsoundness is no problem, though, as long as every Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)

+ +
+ +

Topic 2: box

+ +

box.rkt safely applies unsafe box operations to expressions with Box type.

+ +

Example

+ +
  ;; Type Assumptions
+  (: b (Boxof Symbol))
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (unbox b)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-unbox b)
+ +

Verdict: safe

+ +
+ +

Topic 3: dead-code

+ +

dead-code.rkt uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with if expressions, lambda expressions, and case-lambda; the latter is the most interesting for Transient.

+ +

Example

+ +
  ;; Type Assumptions
+  (: f (-> Symbol Symbol)
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (define f
+    (case-lambda
+      ((s) s)
+      ((s i)
+       (for/list ((_i (in-range i))) s))))
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (define f
+    (case-lambda
+      ((s) s)
+      ((s i)
+       ; dead code, replace with no-op
+       (void))))
+ +

Verdict: unsafe, can change behavior

+ +

The pass infers that some branches of a case-lambda can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the “untyped” branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to (void) and thereby violate the graduality design goal [SNAPL 2015, ICFP 2018] — that is, that adding types should only change behavior by introducing runtime type mismatches.

+ +
+ +

Topic 4: extflonum

+ +

extflonum.rkt safely applies unsafe extflonum operations to expressions with Extflonum type.

+ +

Example

+ +
  ;; Type Assumptions
+  (: e Extflonum)
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (extflabs e)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-extflabs e)
+ +

Verdict: safe

+ +
+ +

Topic 5: fixnum

+ +

fixnum.rkt safely applies unsafe fixnum operations to expressions with Fixnum type.

+ +

Example

+ +
  ;; Type Assumptions
+  (: f Fixnum)
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (exact->inexact f)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-fx->fl f)
+ +

Verdict: safe

+ +
+ +

Topic 6: float-complex

+ +

float-complex.rkt unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.

+ +

Example

+ +
  ;; Type Assumptions
+  (: f (-> Float-Complex Float-Complex Float-Complex))
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (define (f n0 n1)
+    (+ n0 n1))
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (define (f n0 n1)
+    (let* ((unboxed-real-0 (unsafe-flreal-part n0))
+           (unboxed-imag-0 (unsafe-flimag-part n0))
+           (unboxed-real-1 (unsafe-flreal-part n1))
+           (unboxed-imag-1 (unsafe-flimag-part n1))
+           (unboxed-real-2 (unsafe-fl+ (real->double-flonum unboxed-real-0)
+                                       unboxed-real-1))
+           (unboxed-imag-2 (unsafe-fl+ (real->double-flonum unboxed-imag-0)
+                                       unboxed-imag-1)))
+      (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))
+ +

Verdict: safe, with caution

+ +

The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the float-complex pass creates functions that apply unsafe-flreal-part before anything else; to be safe, the pass needs to make sure that Transient checks come first.

+ +
+ +

Topic 7: float

+ +

float.rkt safely applies unsafe flonum operations to expressions with Flonum type and also transforms some random calls to use unsafe-flrandom.

+ +

Example

+ +
  ;; --------------------------------------------------
+  ;; Before Optimization
+  (random)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-flrandom (current-pseudo-random-generator))
+ +

Verdict: safe, but a close call

+ +

Accessing a parameter, as in (current-pseudo-random-generator), is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of unsafe-flrandom.

+ +
+ +

Topic 8: list

+ +

list.rkt safely applies unsafe list operations to list expressions.

+ +

Example

+ +
  ;; Type Assumptions
+  (: lst (List Symbol Symbol))
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (list-ref lst 0)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-list-ref lst 0)
+ +

Verdict: safe, with strong-enough shape checks

+ +

The shape check for a (Listof T) must check for proper lists (via list?); note that the cost of this check depends on the size of incoming values. The shape check for a (List T ...) type must validate the length of incoming values.

+ +
+ +

Topic 9: number

+ +

number.rkt performs simple transformations on Real-valued expressions.

+ +

Example

+ +
  ;; Type Assumptions
+  (: r Real)
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (+ r)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  r
+ +

Verdict: safe

+ +
+ +

Topic 10: pair

+ +

pair.rkt safely applies pair-access operations to (possibly-nested) pairs.

+ +

Example

+ +
  ;; Type Assumptions
+  (: p (Pairof (Pairof Symbol Void) String))
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (cdar p)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-cdr (unsafe-car p))
+ +

Verdict: unsafe

+ +

Transient guarantees the first level of a type, but nothing more. Concretely, Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any and so the unsafe-cdr above is not safe.

+ +
+ +

Topic 11: sequence

+ +

sequence.rkt safely applies unsafe sequence operations to expressions with (Sequenceof T) type.

+ +

Example

+ +
  ;; Type Assumptions
+  (: s String)
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (for ((c s))
+    (void))
+
+  ;; --------------------------------------------------
+  ;; After Optimization (simplified)
+  (for ((c (in-string s)))
+    (void))
+ +

Verdict: safe, with strong enough shape checks (see list and vector)

+ +
+ +

Topic 12: string

+ +

string.rkt safely applies unsafe string operations to expressions with String type. (Note that unsafe-string-ref is only safe when the result is sure to be a Latin–1 character.)

+ +

Example

+ +
  ;; Type Assumptions
+  (: b Bytes)
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (bytes-length b)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-bytes-length b)
+ +

Verdict: safe

+ +
+ +

Topic 13: struct

+ +

struct.rkt safely applies unsafe struct operations to struct expressions, using Typed Racket’s internal registry of struct info.

+ +

Example

+ +
  ;; Type Assumptions
+  (struct open-interval ([lo : Real] [hi : Real]))
+  (: ivl open-interval)
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (open-interval-lo ivl)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-struct-ref ivl 0)
+ +

Verdict: safe

+ +
+ +

Topic 14: unboxed-let

+ +

unboxed-let.rkt cooperates with the float-complex pass by transforming the binding-site of some complex numbers. This pass may change a let-expression into a let-values that expects a real-part and imag-part, and may change a function to expect twice as many arguments — provided the optimizer can find all calls to the function.

+ +

Example

+ +
  ;; Type Assumptions
+  (: k Float-Complex)
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (let ((f (lambda ((n : Float-Complex)) (+ n n))))
+    (f k))
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (let ((f (lambda (real-part-n imag-part-n) ....)))
+    (f (unsafe-flreal-part k) (unsafe-flimag-part k)))
+ +

Verdict: safe, thanks to the (conservative) escape analysis

+ +
+ +

Topic 15: vector

+ +

vector.rkt safely applies vector operations to vector expressions.

+ +

Example

+ +
  ;; Type Assumptions
+  (: v (Vector (Listof Symbol) String))
+  (: lst (Listof Symbol))
+
+  ;; --------------------------------------------------
+  ;; Before Optimization
+  (vector-set! v lst 0)
+
+  ;; --------------------------------------------------
+  ;; After Optimization
+  (unsafe-vector-set! v lst 0)
+ +

Verdict: safe, with strong-enough shape checks

+ +

The check for (Vector T ...) must check the length of incoming values.

+ +
+ +

Summary

+ +

The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (dead-code, pair). One must take care when rewriting a Transient function (float-complex). One may limit our ability to reduce the number of run-time checks in a program (apply). Two others require transient checks whose cost depends on the size of the input values (list, sequence).

+ +

There may be other issues that I missed while reading the optimizer code. If so, I’ll try to remember to update this post.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2020/10/15/transient-answers-old-questions/index.html b/blog/2020/10/15/transient-answers-old-questions/index.html new file mode 100644 index 00000000..079167c5 --- /dev/null +++ b/blog/2020/10/15/transient-answers-old-questions/index.html @@ -0,0 +1,493 @@ + + + + + + Transient Answers Old Questions + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Transient Answers Old Questions

+

+ :: typed racket, transient

+

By: Ben Greenman

+
+ +

Several old questions from the Typed Racket mailing list have new and simple answers under a “transient” Typed Racket.

+ + +
+ +

For the past few months, I’ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the RFC and pull request.

+ +

The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that’s about it. By contrast, types are much stronger in classic Typed Racket.

+ +

Shallow Racket’s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the Racket mailing list. There were a surprising number of questions that went like this:

+ +
+

Q. Hey, I ran a program expecting X to happen, but Y happened instead. Is this a bug?

+

A. No, Typed Racket has to do Y because of its strong types.

+ +

… but changing to shallow types gives the X behavior! Here are their stories.

+ +

Going forward, Deep refers to normal Typed Racket and Shallow refers to Shallow Typed Racket.

+ +
+ +

Higher-Order Value as Any

+ +

Original message : groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ

+ +

On 2018–04–16, mailoo wrote:

+ +
+

I play a little with the “Any” type (due to ‘dynamic-require’ which return Any), and I’m not able to cast them back in a function.

+

I (over) simplify my question with this little program :

+ +
(: p Any) 
+(define (p i) (displayln i)) 
+
+; Here I want to get back my function 
+(define proc (cast p (-> Integer Void))) 
+(proc 2)
+ +
+

but I get this error when I try to execute the function :

+ +
; contract violation 
+; Attempted to use a higher-order value passed as `Any` in untyped code: #<procedure:p>
+ +

What’s going on?

+ +

Deep raises an error because it must enforce the Any type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.

+ +

How’s transient?

+ +

Shallow prints 2 and returns void. No error. Same goes for dynamic-require.

+ +
+ +

Parametric Contract Affects Untyped Code

+ +

Original message : groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ

+ +

On 2019–12–15, John Clements wrote:

+ +
+

It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:

+ +
  #lang typed/racket 
+
+  (require/typed racket/list 
+  [index-of (All (T) ((Listof T) T -> (U False Natural)))]) 
+
+  (index-of '(n s e w) 'n) ;; returns... #f?
+ +
+

In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).

+

I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.

+ +

What’s going on?

+ +

Deep enforces parametricity for All types, and this throws off the equality function that index-of uses internally.

+ +

How’s transient?

+ +

Shallow returns 0.

+ +

ps John, thanks very much for working on Advent of Code and mailing the list!

+ +
+ +

Unable to Protect Opaque Value as Any

+ +

Original message : groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ

+ +

On 2019–12–11, Marc Kaufmann wrote:

+ +
+

I have one file called type-test.rkt with the following

+ +
#lang typed/racket
+
+(require (only-in typed/web-server/http response/xexpr response))
+
+(provide f2)
+
+(: f2 (-> (U response Any)))
+(define (f2)
+  (define x '(body (h1 "Try it")))
+  (: resp response)
+  (define resp (response/xexpr x))
+  resp)
+ +
+

Then I have another untyped file for a servlet:

+ +
#lang racket
+
+(require "type-test.rkt"
+         web-server/servlet
+         web-server/servlet-env)
+
+(define (start req)
+  (f2))
+
+(serve/servlet start
+               #:servlet-regexp #rx""
+               #:launch-browser? #false
+               #:port 8080)
+ +
+

Notice that I am telling [f2] that resp is of type response. Yet, when I run the server with start [….] I get the following result:

+

(f2): Error, see below.

+

The error is:

+ +
f2: broke its own contract
+  any-wrap/c: Unable to protect opaque value passed as `Any`
+  value: #<response>
+  in: the range of
+      (-> Any)
+ +

What’s going on?

+ +

Deep tries to enforce the Any type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.

+ +

How’s transient?

+ +

Shallow does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.

+ +
+ +

Type Inference Installs a Precise Type

+ +

Original message : groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ

+ +

On 2020–02–14, John Clements wrote:

+ +
+

I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.

+

Here’s a program:

+ +
#lang typed/racket 
+
+(define-type Store (Mutable-HashTable Integer Value)) 
+(define-type Value (U Real Boolean String)) 
+
+(define top-store
+  (cast
+    (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f)))
+    Store))
+
+(hash-set! top-store 5 1234)
+ +
+

It fails with this error:

+ +
contract violation
+expected: (or/c (and/c byte? positive?) #t #f)
+given: 1234
+in: the values of
+the 3rd conjunct of
+(and/c hash?
+       hash-mutable?
+       (hash/c exact-integer?
+               (or/c (and/c byte? positive?) #t #f)
+               #:immutable #f))
+ +

What’s going on?

+ +

First off, Deep runs fine after swapping cast for ann.

+ +

Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.

+ +

The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.

+ +

How’s transient?

+ +

Shallow runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.

+ +
+ +

Same-Arity Functions in a Case Lambda

+ +

Original message : groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ

+ +

On 2019–07–05, Ryan Kramer wrote:

+ +
+

In the code below, can maybe-car have the given type [….]?

+ +
#lang typed/racket
+
+(module untyped racket
+  (provide maybe-car)
+  (define (maybe-car x)
+    (cond
+      [(pair? x) (car x)]
+      [else x])))
+
+(require/typed
+ 'untyped
+ [maybe-car (All (a b) (case->
+                        (-> (Pairof a b) a)
+                        (-> a a)))])
+ +
+

[Current error:]

+ +
Type Checker:
+ Type (All (a b) (case-> (-> (Pairof a b) a) (-> a a)))
+  could not be converted to a contract:
+   function type has two cases of arity 1
+ +

What’s going on?

+ +

Deep tries to enforce the type with a Racket or/c contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don’t know which to apply to an incoming function.

+ +

Note, the “Type Checker” error message is much better than what or/c would give on its own.

+ +

How’s transient?

+ +

Shallow simply checks that maybe-car accepts both arities inside the case-> type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.

+ +
+ +

Immutable Type Affects Untyped Code

+ +

Original message : groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ

+ +

On 2020–02–17, Bertrand Augereau wrote:

+ +
+

Hello everybody, I’m trying to gradually type my script to make it a proper app (yes I’m a static-ish guy) and I have an issue (Racket 7.6 CS).

+ +
; racket_mod.rkt:
+#lang racket
+
+(provide (struct-out s))
+(provide list-of-s)
+(provide set-list-of-s!)
+
+(struct s (a))
+(define list-of-s '())
+(define (set-list-of-s! los)
+  (set! list-of-s los))
+ +
; racket_mod_typed.rkt:
+#lang typed/racket
+
+(provide (struct-out s2))
+(provide list-of-s2)
+(provide set-list-of-s2!)
+
+(struct s2 ([a : Natural]))
+(define list-of-s2 '())
+(define (set-list-of-s2! [los : (Listof s2)])
+  (set! list-of-s2 los))
+ +
; racket_main.rkt:
+#lang racket
+
+(require "racket_mod.rkt")
+(require "racket_mod_typed.rkt")
+
+(define los (list (s 1) (s 2)))
+(set-list-of-s! los)
+(displayln list-of-s)
+
+(define los2 (list (s2 1) (s2 2)))
+(set-list-of-s2! los2)
+(displayln list-of-s2)
+ +
+

list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)

+ +

What’s going on?

+ +

Deep enforces the type of list-of-s2 with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.

+ +

Here’s a step-by-step breakdown:

+ +
    +
  1. the typed module creates an empty list-of-s2
    2. +
  2. the main module imports the list and receives a new copy
    3. +
  3. the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable
    4. +
  4. the main module reads from its copy — and it’s still empty
+ +

How’s transient?

+ +

Shallow lets the original list travel to untyped code. There are no contracts in the way.

+ +

Discussion

+ +

Wow! It’s great to see that Shallow Racket works “as expected” on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.

+ +

If you have a similar experience with a deep-types error, let me know.

+ +

Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error — and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.

+ +

Shallow Typed Racket is coming soon. Follow the pull request or watch the Racket release notes for news.

+ + + + + +

Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2020/11/12/transient-for-optional-and-keyword-functions/index.html b/blog/2020/11/12/transient-for-optional-and-keyword-functions/index.html new file mode 100644 index 00000000..c30c58ca --- /dev/null +++ b/blog/2020/11/12/transient-for-optional-and-keyword-functions/index.html @@ -0,0 +1,516 @@ + + + + + + Transient for Optional and Keyword Functions + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Transient for Optional and Keyword Functions

+

+ :: typed racket, transient

+

By: Ben Greenman

+
+ +

A short adventure into the depths of optional and/or keyword functions in Racket.

+ + +
+ +

Transient, or rather the Transient semantics for a mixed-typed language, is one way to let statically-typed code safely interact with untyped code. You can read all about it in Michael Vitousek’s 2019 dissertation or my 2020 dissertation, and you can see how it compares to other mixed-typed semantics here. The idea is to give up on behavioral type guarantees and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called shape checks; for example:

+ +
    +
  • if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;
    • +
  • if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and
    • +
  • if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.
+ +

Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type T into a shape check, and we have a function with type T that needs to check its inputs. The question is how to actually do the check in Racket v7.9.

+ +

In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if T = (-> Symbol Boolean) then we need to check the shape symbol? of the domain type Symbol:

+ +
;; source code
+(: f (-> Symbol Boolean))
+(define (f sym)
+  (eq? sym 'hola))
+
+;; ===>
+
+;; imaginary (but realistic) rewritten code
+(define (f sym)
+  (assert sym symbol?)
+  (eq? sym 'hola))
+ +

A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type T accepts 1 to 3 inputs:

+ +
;; source code
+(: g (->* [#:a Boolean] [Symbol #:c Void] Symbol))
+(define (g #:a a [b 'b] #:c [c #f])
+  (if a b (if c 'left 'right)))
+
+;; ===>
+
+;; imaginary, unrealistic rewritten code
+(define (g #:a a [b 'b] #:c [c #f])
+  (assert a boolean?)
+  (assert b symbol?)
+  (assert c void?)
+  (if a b (if c 'left 'right)))
+ +

Good — we basically know what we want. If the Racket core language had optional and keyword functions, then we’d be done.

+ +

But no, Racket expands these optional/keyword functions into primitive lambda and case-lambda forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.

+ +

Let’s keep digging.

+ +

From now on, “Shallow” or “Shallow TR” refers to my implementation of Transient for Typed Racket (TR). We’ll talk about Shallow instead of “Transient” in case future work reveals a better way to implement the Transient idea.

+ +

False Start: Follow the Type

+ +

Beware — Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function g above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional #:c keyword lead to a shape-check failure because the variable c gets the default value #f instead of a void value. Concretely, the third assert from above fails:

+ +
(define (g #:a a [b 'b] #:c [c #f])
+  ....
+  (assert c void?) ;; fails if c is the #f default value
+  ....)
+ +

The problem arises from subtyping. According to the annotations, the function g has an external type that is less precise than the internal type that validates the function body:

+ +
;; external type T
+(: g (->* [#:a Boolean] [Symbol #:c Void] Symbol))
+
+;; internal type T2, subtype of external (T2 <: T), validates body
+(: g (->* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))
+ +

Thanks to this external / internal distinction, the following easy rewrite idea, Solution 0, fails. Despite the failure, this first solution is a useful starting point for a success.

+ +

Solution 0, Step 1: Mimic the Typechecker

+ +

Shallow TR uses the same type checker as classic Deep TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.

+ +

Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.

+ +

Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (typecheck/tc-expr-unit.rkt). The omitted code (….) does actual type checking:

+ +
(define (tc-expr/check/internal form expected-type)
+  ....
+  (syntax-parse form
+    #:literal-sets (kernel-literals tc-expr-literals)
+    ....
+    [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^)
+    ....]
+    [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^)
+    ....]
+ +

Ok! Those two patterns say a lot about the expansion of optional and keyword functions:

+ +
    +
  1. Both forms expand to a let-values that binds one function fun.
    2. +
  2. TR uses the syntax classes kw-lambda^ and opt-lambda^ to tell these particular let-values apart from others.
+ +

Shallow TR can use exactly these patterns to recognize optional/keyword functions.

+ +

Solution 0, Step 2: Parse the Domain Type

+ +

Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function’s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:

+ +
    [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^)
+    ....]
+ +

First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The type-of function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, fun, has a type. Great.

+ +

Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:

+ +
    +
  • The type of fun could be a straightforward Fun type, but it could also be a: DepFun type, or Poly type, or PolyDots type, or even a Union type.
    • +
  • Each part of the domain type corresponds to one parameter of the fun expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in fun, or the mandatory keywords?
+ +

Solution 0, Step 3: Insert a Shape Check

+ +

Once we have the target fun expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse fun to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting fun goes something like this:

+ +
(syntax-parse fun
+  [(#%plain-lambda formals . body)
+   #:with (shape-check ...)
+          (make-shape-checks #'formals (type-of fun))
+   #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])
+ +

The rewritten function executes shape checks immediately, and then proceeds with the body after validating each actual parameter.

+ +

On the Trail: optkey Expansion

+ +

Our Solution 0 fails because the type of the fun expression that it gets from the type-checked code is an external type. In terms of the g function from above, Solution 0 uses the type Void instead of the internal type (U Void #f) to check the c parameter. To get internal types, we need to look closer at fun and the rest of the optional/keyword expansion.

+ +

Let’s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.

+ +

If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function X create a module test.rkt like this:

+ +
#lang racket/base
+
+(define _ignore
+  (let ()
+    X
+    (void)))
+ +

Invoke the expander with raco expand test.rkt > test.rkt.txt and explore the generated .txt file.

+ +

Example 1: mandatory keyword

+ +

The source is a function with one mandatory positional argument and one optional positional argument.

+ +
(lambda (x [y 0])
+  (+ x y))
+ +

Expansion generates a case-lambda that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function F that expects two arguments, resolves defaults in a different way, and executes the function body.

+ +
(let-values (((F)
+              (lambda (x2 y1)
+                (let-values (((x) x2))
+                  (let-values (((y) (if '#f '0 y1)))
+                    (let-values () (#%app + x y)))))))
+  (case-lambda
+   ((x) (#%app F x '0))
+   ((x y1) (#%app F x y1))))
+ +

Note: the expression (if ’#f ’0 y1) in the generated F function is equal to y1 alone. In general, the if is for default expressions. (Unlike Python, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a #f test. A general-purpose optimizer can remove this test before the code runs.

+ +

Example 2:

+ +

The source is a function with one mandatory positional argument and one mandatory keyword argument:

+ +
(lambda (x #:y y)
+  (+ x y))
+ +

Expansion generates several functions:

+ +
    +
  • F0 expects a plain list of arguments and executes the source function’s body
    • +
  • F1 expects a list of keywords, a list of arguments, and a final argument. The purpose of F1 is to organize a call to F0.
    • +
  • lifted/2 is the constructor for a generated struct type. Other functions help the struct call F1. Nevermind the details; I don’t fully understand them either.
+ +

The important piece for Shallow TR is the F0 function because the goal of rewriting is to protect the original function body against untyped inputs.

+ +
(let-values (((F0)
+              (lambda (y1 x3)
+                (let-values (((x) x3))
+                  (let-values (((y) y1))
+                    (let-values () (#%app + x y)))))))
+  (let-values (((F1)
+                (lambda (given-kws given-args x3)
+                  (let-values (((y1) (#%app car given-args)))
+                    (#%app F0 y1 x3)))))
+    (#%app
+     lifted/2
+     (lambda (given-kws given-argc)
+       (if (#%app = given-argc '3)
+         (let-values (((l2571) given-kws))
+           (if (#%app pair? l2571)
+             (if (#%app eq? (#%app car l2571) '#:y)
+               (#%app null? (#%app cdr l2571))
+               '#f)
+             '#f))
+         '#f))
+     (case-lambda
+      ((given-kws given-args x)
+       (#%app F1 given-kws given-args x)))
+     '(#:y)
+     '(#:y))))
+ +

Example 3:

+ +

The source is a function with one mandatory positional argument and one optional keyword argument:

+ +
(lambda (x #:y [y 0])
+  (+ x y))
+ +

Expansion again generates several functions:

+ +
    +
  • F0 expects a plain list of arguments, resolves the optional default, and executes the source function’s body
    • +
  • F1 calls F0
    • +
  • At the bottom, there are two case-lambda functions that call F1
+ +

Again, the F0 function is the focal point for Shallow TR rewriting.

+ +
(let-values (((F0)
+              (lambda (y1 x3)
+                (let-values (((x) x3))
+                  (let-values (((y) (if '#f '0 y1)))
+                    (let-values () (#%app + x y)))))))
+  (let-values (((F1)
+                (lambda (given-kws given-args x3)
+                  (let-values (((y2) (#%app pair? given-kws)))
+                    (let-values (((y1)
+                                  (if y2 (#%app car given-args) '0)))
+                      (#%app F0 y1 x3))))))
+    (#%app
+     make-optional-keyword-procedure
+     (lambda (given-kws given-argc)
+       (if (#%app = given-argc '3)
+         (let-values (((l1571) given-kws))
+           (let-values (((l1571)
+                         (if (#%app null? l1571)
+                           l1571
+                           (if (#%app eq? (#%app car l1571) '#:y)
+                             (#%app cdr l1571)
+                             l1571))))
+             (#%app null? l1571)))
+         '#f))
+     (case-lambda
+      ((given-kws given-args x)
+       (#%app F1 given-kws given-args x)))
+     null
+     '(#:y)
+     (case-lambda
+      ((x) (#%app F1 null null x))))))
+ +

Solution: The Shallow TR Rewrite Strategy

+ +

All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a let-values form:

+ +
(let-values (((f) fun)) . body)
+ +

Furthermore, the generated fun is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the fun from Example 3 again; it has formal parameters for the keyword arg. and the mandatory arg., and one let-values to resolve each parameter:

+ +
  (lambda (y1 x3)
+    (let-values (((x) x3))
+      (let-values (((y) (if '#f '0 y1)))
+        (let-values () (#%app + x y)))))
+ +

Another experiment with type-of shows that the right-hand side of each let-values has an internal type annotation. Excellent! Both (type-of x3) and (type-of (if ’#f ’0 y1)) are the right types for shape checks. Shallow TR can:

+ +
    +
  • inspect the let-values one-by-one;
    • +
  • convert the type of each right-hand expression to a shape predicate; and
    • +
  • rewrite each right-hand expr into (assert expr shape?).
+ +

This should work! In fact, we can do slightly better:

+ +
    +
  • when the right-hand expression is a conditional (if test default-expr supplied-arg)
    • +
  • then Shallow only needs to check the supplied arg: (if test default-expr (assert supplied-arg shape?))
+ +

Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.

+ +

A Problem with Methods and a Bugfix

+ +

Currently, Shallow TR rewrites optional and keyword functions using the let-values plan described above. Each formal parameter has one let-values binding, and the type on each bound expression defines the shape check.

+ +

Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we fixed the class expander.

+ +

First, here is a class with one method that runs correctly. The method f accepts an optional positional argument x; the default value of x is the current value of the field my-num (fields are mutable):

+ +
(define c0%
+  (class object%
+    (super-new)
+    (field (my-num 2))
+    (define/public (f [x my-num])
+      (+ x x))))
+ +

Second, here is a similar method that fails. This time, the default is an immediate value 2:

+ +
(define c1%
+  (class object%
+    (super-new)
+    (define/public (f [x 2])
+      (+ x x))))
+ +

Running a call (send o1 f) used to raise a shape-check failure about a strange value:

+ +
+

shape error: Expected a real number, got #<unsafe-undefined>

+ +

What is going on?

+ +

It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:

+ +
(lambda (x [y z])
+  (+ x y))
+ +

Expansion generates a function F0 that checks for the undefined value, and an outer case-lambda that supplies undefined when the default is needed:

+ +
(let-values (((F0)
+              (lambda (x2 y1)
+                (let-values (((x) x2))
+                  (let-values (((y)
+                                (if (#%app eq? y1 unsafe-undefined)
+                                  z
+                                  y1)))
+                    (let-values () (#%app + x y)))))))
+  (case-lambda
+   ((x) (#%app F0 x unsafe-undefined))
+   ((x y1) (#%app F0 x y1))))
+ +

That’s the normal way that unsafe-undefined shows up: the expander for optional/keyword functions looks for default expressions vs. default values and uses the undefined value for expressions.

+ +

Three other facts conspired to make the problem with optional methods:

+ +
    +
  1. Typed Racket also looks for default expressions vs. default values (search for immediate-default here). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the unsafe-undefined value (search for -Unsafe-Undefined here (kw) and here (opt)).
    2. +
  2. The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.
    3. +
  3. Shallow TR pushes default expression checks (if test default-expr supplied-arg) to the supplied-arg instead of wrapping the whole if form.
+ +

In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound—but harmless because the false assumption was guarded by an if test for unsafe-undefined. Running Shallow TR revealed the unsoundness with its eager shape check.

+ +

Again, the resolution was to fix the class expander (racket/racket #3182). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.

+ +

Lessons

+ +
    +
  1. Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.
    2. +
  2. Digging into the expansion is sometimes necessary. There are at least three places that do so—the class expander, TR, and Shallow TR—and unfortunately they all need to cooperate.
    3. +
  3. The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of my dissertation lists them all.
+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2020/12/23/deep-and-shallow-types/index.html b/blog/2020/12/23/deep-and-shallow-types/index.html new file mode 100644 index 00000000..ecb38612 --- /dev/null +++ b/blog/2020/12/23/deep-and-shallow-types/index.html @@ -0,0 +1,186 @@ + + + + + + Deep and Shallow Types + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Deep and Shallow Types

+

+ :: dissertation, migratory typing

+

By: Ben Greenman

+
+ +

I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:

+ + + +

To the PRL: thanks for a wonderful 6.5 years.

+ + +

Abstract

+ +
+

The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound’’ mixed-typed languages from others that enforce type obligations locally rather than globally.

+

In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.

+

My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.

+

I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.

+ +

Next year, I’ll be a CI Fellow at Brown.

+

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/index.html b/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/index.html new file mode 100644 index 00000000..3f102e9f --- /dev/null +++ b/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/index.html @@ -0,0 +1,405 @@ + + + + + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript)

+

+ :: visr, clojure, clojurescript, interactive syntax

+

By: Leif Andersen

+
+ +

+

+ +

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

+ +
Graphical network embedded in text +

Graphical network embedded in text

+ +

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

+ +
    +
  • version control, such as git, works with interactive-syntax;
    • +
  • programs using interactive-syntax can be written and edited with your favorite text editor or IDE;
    • +
  • cut/copy/paste works with interactive-syntax using your operating system’s native clipboard;
    • +
  • code analysis tools, like diff and refactor, still work with interactive-syntax; and
    • +
  • you can use interactive-syntax in any language or environment that supports language-oriented programming.
+ +

To learn more about interactive-syntax, watch this video or read the accompanying paper.

+ + + +

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

+ +

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

+ + +

Getting started with VISr

+ +

Start by going to visr.pl, which is a web-based IDE that directly supports VISrs. Once in the IDE, press Insert VISr to place a VISr at the current cursor position. This VISr contains two buttons:

+ +
    +
  • clicking the first displays the VISr’s visual representation, and
    • +
  • clicking the second shows its textual representation.
+ +
VISr +

VISr

+ +

Opening the code shows that the new VISr is an instance of visr.core/empty-visr, a default VISr provided by the IDE. This VISr expects a map with the key :message to display in the visual view. Changing the value associated with :message changes what is displayed, in this case “Endless Possibility”:

+ +
Open Visr +

Open Visr

+ +

Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:

+ +
+ + + + +
+
+ 
1
+2
 +
+ 
^{:editor visr.core/empty-visr}(visr.core/empty-visr+elaborate 
+                                 {:message "Endless Possibility"})
+
+
+
+ +

This operation works in reverse too. Writing out similar text and pasting it into visr.pl yields its visual representation.

+ +

Making a new VISr

+ +

The defvisr form creates a VISr type. This form expects two methods:

+ +
    +
  1. a render method that provides visualization and interaction when code is edited, and
    2. +
  2. an elaborate/elaborate-fn method that gives the VISr compile-time and run-time semantics.
+ +

The following is the signature for a simple VISr type:

+ +
+ + + + +
+
+ 
1
+2
+3
+4
+5
 +
+ 
(ns example.core)
+
+(defvisr Counter
+  (elaborate-fn [this] "TODO-elaborate")
+  (render [this] "TODO-render"))
+
+
+
+ +

This example uses elaborate-fn, a simplified version of elaborate that gives the VISr the same semantics as a function application. It also allows the defvisr form to work in the same file as the VISr itself.

+ +
Example of elaborate-fn semantics +

Example of elaborate-fn semantics

+ +

The Render Method for Edit-Time Semantics

+ +

The render method is given the VISr state as an atom; updating this atom also updates the code to reflect the new state. The return value for render must be a Reagent form that is the visual view for the VISr. A render method for a counter VISr might look as follows:

+ +
+ + + + +
+
+ 
1
+2
 +
+ 
(render [this]
+  [:button {:on-click #(swap! this inc)} @this])
+
+
+
+ +

And in action:

+ +
Simple Count Example +

Simple Count Example

+ +

This VISr doesn’t match the theme of the page; it also requires the state to be a single number. Using React Bootstrap and Reagent cursors fixes both of these issues:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
 +
+ 
(ns example.core
+  (:require [reagent.core :refer [cursor]]
+            [react-bootstrap :refer [Button]]))
+
+(defvisr Counter
+  (elaborate-fn [this] "TODO-elaborate")
+  (render [this]
+    (let [count (cursor this [:count])]
+      (when-not @count (reset! count 0))
+      [:> Button {:on-click #(swap! count inc)} @count])))
+
+
+
+ +

Elaboration and Run-Time Semantics

+ +

The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of elaborate-fn, the VISr semantics takes the form of a function application:

+ +
+ + + + +
+
+ 
1
 +
+ 
(elaborate-fn [{:keys [count]}] count)
+
+
+
+ +

This elaborate method expects a dictionary with the key :count and returns the value associated with that key. It makes use of ClojureScript’s Destructuring for brevity. The following code is equivalent:

+ +
+ + + + +
+
+ 
1
 +
+ 
(elaborate-fn [this] (get this :count))
+
+
+
+ +

Putting it all together

+ +

The final result is:

+ +
+ + + + +
+
+ 
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
 +
+ 
(ns test.core
+  (:require [reagent.core :refer [cursor]]
+            [react-bootstrap :refer [Button]]))
+
+(defvisr Counter
+  (elaborate-fn [{:keys [count]}] count)
+  (render [this]
+    (let [count (cursor this [:count])]
+      (when-not @count (reset! count 0))
+      [:> Button {:on-click #(swap! count inc)} @count])))
+
+
+
+ +

Here is the VISr in action:

+ +
Full Count Example +

Full Count Example

+ +

That’s all there is to it. From here, you can go to visr.pl to make your own programs using VISr. You can also take this survey, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to the visr project page.

+ +

Thanks for reading, happy coding!

+

+

+

+

+ +
+
+ + + +
+
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git "a/blog/2022/02/16/-r\314\214-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/index.html" "b/blog/2022/02/16/-r\314\214-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/index.html" new file mode 100644 index 00000000..88b79a99 --- /dev/null +++ "b/blog/2022/02/16/-r\314\214-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/index.html" @@ -0,0 +1,167 @@ + + + + + + [Ř Overview I (cross-post)](https://www.o1o.ch/lab/entry/1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc.html) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Ř Overview I (cross-post)

+

+ :: Ř JIT

+

By: Olivier Flückiger

+
+ +

+

+

+

+ +
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/feeds/1st-blog-post.atom.xml b/blog/feeds/1st-blog-post.atom.xml new file mode 100644 index 00000000..31138d7f --- /dev/null +++ b/blog/feeds/1st-blog-post.atom.xml @@ -0,0 +1,39 @@ + + + PRL Blog: Posts tagged '1st blog post' + + + urn:http-prl-ccs-neu-edu:-blog-tags-1st-blog-post-html + 2016-04-29T14:50:29Z + + Welcome to the PRL blog + + urn:http-prl-ccs-neu-edu:-blog-2016-04-29-welcome-to-the-prl-blog + 2016-04-29T14:50:29Z + 2016-04-29T14:50:29Z + + Ben Greenman + +<p>Greetings, ground rules, hopes, dreams, and notes for contributors. Welcome aboard.</p> +<!-- more--> + +<p>Earlier this year, the Programming Research Lab (PRL) was blessed with a new postdoc: <a href="http://gallium.inria.fr/~scherer/">Gabriel Scherer</a> from INRIA Paris-Rocquencourt, France. Ever since Gabriel arrived things have been changing here in Boston. We now have homemade bread on the first Tuesday of every month, <a href="https://en.wikipedia.org/wiki/Orange_flower_water">orange water</a> crepes after holidays, and someone new to go out for bubble tea with in between. All that and an enthusiastic colleague and researcher.</p> + +<p>In his spare time between lobbying the CS department for an espresso machine and building multi-language compilers, Gabriel is also a champion of open access. Hence this blog, a window into the life and times of PRL students made possible by Gabriel&rsquo;s tactical prodding and careful delegation of responsibilities. Anything you might read about in a rejected conference paper or hear over coffee is fair game here: the goal is to give the wide world a glimpse of our lab and people.</p> + +<h2 id="for-contributors">For Contributors</h2> + +<p>These pages are generated using Greg Hendershott&rsquo;s <a href="https://github.com/greghendershott/frog">frog</a> static website generator. To create a new post:</p> + +<ol> + <li>Clone or fork the <a href="https://github.com/nuprl/nuprl.github.io">nuprl.github.io</a> repository</li> + <li>Check out a new git branch for your post</li> + <li>Run <code>cd blog; raco frog -n "TITLE"</code> to build a template for a new post</li> + <li>Add content to the new markdown file (under <code>_src/posts</code>)</li> + <li>Rebuild the blog with <code>raco frog -b</code></li> + <li>Run <code>cd ..; raco frog -p</code> to start a web server and view your changes at <a href="http://localhost:3000/">http://localhost:3000/</a></li> + <li>Send a pull request to the <a href="https://github.com/nuprl/nuprl.github.io">nuprl.github.io</a> repo</li></ol> + +<p>An open pull request is the best place to ask questions about the formatting or content of a post. We promise that within a few days of opening a PR someone with push access will reply with feedback or merge the request.</p> + +<p>Contributions are open to anyone: current labmates, alumni, friends from the Racket mailing list, and even recovering C programmers. One should have a strong connection to Northeastern or our research, but even that is not strictly necessary. Visitors are always welcome to the PRL.</p> \ No newline at end of file diff --git a/blog/feeds/1st-blog-post.rss.xml b/blog/feeds/1st-blog-post.rss.xml new file mode 100644 index 00000000..605dca5a --- /dev/null +++ b/blog/feeds/1st-blog-post.rss.xml @@ -0,0 +1,39 @@ + + + + PRL Blog: Posts tagged '1st blog post' + PRL Blog: Posts tagged '1st blog post' + http://prl.ccs.neu.edu/blog/tags/1st-blog-post.html + Fri, 29 Apr 2016 14:50:29 UT + Fri, 29 Apr 2016 14:50:29 UT + 1800 + + Welcome to the PRL blog + http://prl.ccs.neu.edu/blog/2016/04/29/welcome-to-the-prl-blog/?utm_source=1st-blog-post&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-04-29-welcome-to-the-prl-blog + Fri, 29 Apr 2016 14:50:29 UT + Ben Greenman + +<p>Greetings, ground rules, hopes, dreams, and notes for contributors. Welcome aboard.</p> +<!-- more--> + +<p>Earlier this year, the Programming Research Lab (PRL) was blessed with a new postdoc: <a href="http://gallium.inria.fr/~scherer/">Gabriel Scherer</a> from INRIA Paris-Rocquencourt, France. Ever since Gabriel arrived things have been changing here in Boston. We now have homemade bread on the first Tuesday of every month, <a href="https://en.wikipedia.org/wiki/Orange_flower_water">orange water</a> crepes after holidays, and someone new to go out for bubble tea with in between. All that and an enthusiastic colleague and researcher.</p> + +<p>In his spare time between lobbying the CS department for an espresso machine and building multi-language compilers, Gabriel is also a champion of open access. Hence this blog, a window into the life and times of PRL students made possible by Gabriel&rsquo;s tactical prodding and careful delegation of responsibilities. Anything you might read about in a rejected conference paper or hear over coffee is fair game here: the goal is to give the wide world a glimpse of our lab and people.</p> + +<h2 id="for-contributors">For Contributors</h2> + +<p>These pages are generated using Greg Hendershott&rsquo;s <a href="https://github.com/greghendershott/frog">frog</a> static website generator. To create a new post:</p> + +<ol> + <li>Clone or fork the <a href="https://github.com/nuprl/nuprl.github.io">nuprl.github.io</a> repository</li> + <li>Check out a new git branch for your post</li> + <li>Run <code>cd blog; raco frog -n "TITLE"</code> to build a template for a new post</li> + <li>Add content to the new markdown file (under <code>_src/posts</code>)</li> + <li>Rebuild the blog with <code>raco frog -b</code></li> + <li>Run <code>cd ..; raco frog -p</code> to start a web server and view your changes at <a href="http://localhost:3000/">http://localhost:3000/</a></li> + <li>Send a pull request to the <a href="https://github.com/nuprl/nuprl.github.io">nuprl.github.io</a> repo</li></ol> + +<p>An open pull request is the best place to ask questions about the formatting or content of a post. We promise that within a few days of opening a PR someone with push access will reply with feedback or merge the request.</p> + +<p>Contributions are open to anyone: current labmates, alumni, friends from the Racket mailing list, and even recovering C programmers. One should have a strong connection to Northeastern or our research, but even that is not strictly necessary. Visitors are always welcome to the PRL.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Aaron-Weiss.atom.xml b/blog/feeds/Author-Aaron-Weiss.atom.xml new file mode 100644 index 00000000..82eb726e --- /dev/null +++ b/blog/feeds/Author-Aaron-Weiss.atom.xml @@ -0,0 +1,16 @@ + + + PRL Blog: Posts tagged 'Author: Aaron Weiss' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Aaron-Weiss-html + 2017-06-09T13:36:56Z + + [Bridging the System Configuration Gap (Cross-Post)](https://aaronweiss.us/posts/2017-06-05-bridging-the-system-configuration-gap.html) + + urn:http-prl-ccs-neu-edu:-blog-2017-06-09-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html + 2017-06-09T13:36:56Z + 2017-06-09T13:36:56Z + + Aaron Weiss + \ No newline at end of file diff --git a/blog/feeds/Author-Aaron-Weiss.rss.xml b/blog/feeds/Author-Aaron-Weiss.rss.xml new file mode 100644 index 00000000..087226c2 --- /dev/null +++ b/blog/feeds/Author-Aaron-Weiss.rss.xml @@ -0,0 +1,16 @@ + + + + PRL Blog: Posts tagged 'Author: Aaron Weiss' + PRL Blog: Posts tagged 'Author: Aaron Weiss' + http://prl.ccs.neu.edu/blog/tags/Author-Aaron-Weiss.html + Fri, 09 Jun 2017 13:36:56 UT + Fri, 09 Jun 2017 13:36:56 UT + 1800 + + [Bridging the System Configuration Gap (Cross-Post)](https://aaronweiss.us/posts/2017-06-05-bridging-the-system-configuration-gap.html) + http://prl.ccs.neu.edu/blog/2017/06/09/-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html/?utm_source=Author-Aaron-Weiss&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-09-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html + Fri, 09 Jun 2017 13:36:56 UT + Aaron Weiss + \ No newline at end of file diff --git a/blog/feeds/Author-Alexi-Turcotte.atom.xml b/blog/feeds/Author-Alexi-Turcotte.atom.xml new file mode 100644 index 00000000..3a1b4d6a --- /dev/null +++ b/blog/feeds/Author-Alexi-Turcotte.atom.xml @@ -0,0 +1,45 @@ + + + PRL Blog: Posts tagged 'Author: Alexi Turcotte' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Alexi-Turcotte-html + 2019-03-09T14:40:16Z + + PLISS: Learn About PL Implementation in a Castle + + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + 2019-03-09T14:40:16Z + 2019-03-09T14:40:16Z + + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> \ No newline at end of file diff --git a/blog/feeds/Author-Alexi-Turcotte.rss.xml b/blog/feeds/Author-Alexi-Turcotte.rss.xml new file mode 100644 index 00000000..2ea2addb --- /dev/null +++ b/blog/feeds/Author-Alexi-Turcotte.rss.xml @@ -0,0 +1,45 @@ + + + + PRL Blog: Posts tagged 'Author: Alexi Turcotte' + PRL Blog: Posts tagged 'Author: Alexi Turcotte' + http://prl.ccs.neu.edu/blog/tags/Author-Alexi-Turcotte.html + Sat, 09 Mar 2019 14:40:16 UT + Sat, 09 Mar 2019 14:40:16 UT + 1800 + + PLISS: Learn About PL Implementation in a Castle + http://prl.ccs.neu.edu/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/?utm_source=Author-Alexi-Turcotte&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + Sat, 09 Mar 2019 14:40:16 UT + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> \ No newline at end of file diff --git a/blog/feeds/Author-Artem-Pelenitsyn.atom.xml b/blog/feeds/Author-Artem-Pelenitsyn.atom.xml new file mode 100644 index 00000000..e4fe9bf5 --- /dev/null +++ b/blog/feeds/Author-Artem-Pelenitsyn.atom.xml @@ -0,0 +1,131 @@ + + + PRL Blog: Posts tagged 'Author: Artem Pelenitsyn' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Artem-Pelenitsyn-html + 2017-06-16T11:38:25Z + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Programming Language Conference in Russia + + urn:http-prl-ccs-neu-edu:-blog-2017-05-24-programming-language-conference-in-russia + 2017-05-24T12:25:17Z + 2017-05-24T12:25:17Z + + Artem Pelenitsyn + +<p>In April 3&mdash;5 I took part into a Russian conference exclusively devoted to programming languages: <a href="http://plc.sfedu.ru/">Programming Languages and Compilers</a> (<a href="https://translate.google.com/translate?sl=auto&amp;tl=en&amp;js=y&amp;prev=_t&amp;hl=en&amp;ie=UTF-8&amp;u=http%3A%2F%2Fplc.sfedu.ru%2F&amp;edit-text=&amp;act=url">Google.Translated version of the site</a>). I was a member of organizing committee and had a paper there.</p> + +<p>This is the first conference in Russia highly focused on our area of PL. At least for the last several decades (I believe, there were conferences of the kind back in USSR). The conference was devoted to the memory of prominent Soviet PL-researcher from Rostov-on-Don, Adolf Fuksman who worked on ideas quite similar to what we know as the aspect-oriented programming back in the 70-s.</p> +<!-- more--> + +<p>We designed and implemented the conference with my colleagues from I.I.Vorovich institute of Mathematics, Mechanics and Computer Science, Southern Federal University (<a href="https://www.google.com/maps/place/Rostov-on-Don,+Rostov+Oblast,+Russia/@49.8345629,18.9321123,4.5z/data=!4m5!3m4!1s0x40e3c777c3b4b6ef:0x8248b451e48b4d04!8m2!3d47.2357137!4d39.701505">Rostov-on-Don</a>, Russia). We aimed at gathering as much PL-researchers and enthusiasts from academia in Russia as we could. One of the consequences of the aim was a decision to run the conference in Russian. Though we missed expertise from our non-Russian speaking colleagues, we got thorough participation from all over Russia:</p> + +<blockquote> + <p>Saint-Petersburg, Moscow, Novosibirsk, Krasnoyarsk, Ekaterinburg, Kazan, etc.</p></blockquote> + +<p>I only mention here the cities with more than 1 mil. population in decreasing in number of conference participants order (and excluding Rostov itself, of course).</p> + +<p>I particularly liked talks by invited speakers. When searching for ones, we targeted Russians who work at prominent universities and/or have some visibility at the international level. We ended up with two researchers: <a href="http://ilyasergey.net/">Ilya Sergey</a> (University College of London) and <a href="http://www.macs.hw.ac.uk/~ek19/">Ekaterina Komendantskaya</a> (Heriot-Watt U., Edinburg, UK). Interestingly, their talks were quite close to each other:</p> + +<ul> + <li>I. Sergey, Dependent Types for verification of real-world programs,</li> + <li>E. Komendantskaya, Automated Theorem Proving for Type Inference, Constructively.</li></ul> + +<p>Both of them talked about types as a logic tool to ensure program correctness.</p> + +<p>Biggest opening in this conference for me was a team from Languages Tools Laboratory of <a href="https://en.wikipedia.org/wiki/JetBrains">JetBrains</a>. Surely, you heard about JB: either about their famous IDE, IntelliJ IDEA, or the Kotlin programming language (which, by the way, <a href="https://blog.jetbrains.com/kotlin/2017/05/kotlin-on-android-now-official/">is endorsed</a> for Android development these days). You may also have noticed that JB become sponsors of ECOOP and OPLSS this year. So we had a whole team of researchers from Saint-Petersburg office of JB. Among their topics: <code>OCaml</code> embedding of <code>miniKanren</code> (some results was presented on ML Workshop 2016), parser combinator libraries for <code>OCaml</code> and constrained graph querying (this is not specifically a PL problem, see <a href="https://arxiv.org/abs/1502.02242">arXiv:1502.02242</a> for details).</p> + +<p>Otherwise the spectrum of topics presented on the conference was quite broad, here are some:</p> + +<ul> + <li>Static analysis with PVS-studio (a sponsor talk),</li> + <li>supercompilation (a talk by researchers from Pereslavl-Zalesskiy, where the topic is actively developed for decades),</li> + <li>C++ and beyond (by a member of the ISO C++ committee),</li> + <li>architecture-agnostic parallel programming languages and compilation techniques for parallel architectures,</li> + <li>game semantics and ontologies for PL semantics,</li> + <li>program analysis,</li> + <li>compiler architectures.</li></ul> + +<p>Full program with links to slides in Russian is available <a href="https://docs.google.com/spreadsheets/d/11QiFUqJG_NiBHVUfji_6-FiqP3aQWmdDBN13abM32nY/edit?usp=sharing">here</a>.</p> + +<p>Let me mention my submission: that was a joint work with a student of mine on exploring design space for parser combinator libraries using programming language with direct support of effect system, namely <a href="http://popl17.sigplan.org/event/popl-2017-papers-do-be-do-be-do">Frank</a>. <a href="http://staff.mmcs.sfedu.ru/%7Ejuliet/index.en.html">Julia Belyakova</a> also participated in the conference with her work on Coq-certified interpreter for an extension of lambda-calculus with concept-parameters (module-like kind of thing). The follow-up of that work is accepted for FTfJP workshop this year. You can also listen to her on the topic at the <a href="http://www.nepls.org/Events/30/">NEPLS</a> next week.</p> + +<p>I hope that we will find sources, time, and, most important, high quality submissions for PLC&ndash;2018.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Artem-Pelenitsyn.rss.xml b/blog/feeds/Author-Artem-Pelenitsyn.rss.xml new file mode 100644 index 00000000..9c8d2792 --- /dev/null +++ b/blog/feeds/Author-Artem-Pelenitsyn.rss.xml @@ -0,0 +1,121 @@ + + + + PRL Blog: Posts tagged 'Author: Artem Pelenitsyn' + PRL Blog: Posts tagged 'Author: Artem Pelenitsyn' + http://prl.ccs.neu.edu/blog/tags/Author-Artem-Pelenitsyn.html + Fri, 16 Jun 2017 11:38:25 UT + Fri, 16 Jun 2017 11:38:25 UT + 1800 + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=Author-Artem-Pelenitsyn&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Programming Language Conference in Russia + http://prl.ccs.neu.edu/blog/2017/05/24/programming-language-conference-in-russia/?utm_source=Author-Artem-Pelenitsyn&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-24-programming-language-conference-in-russia + Wed, 24 May 2017 12:25:17 UT + Artem Pelenitsyn + +<p>In April 3&mdash;5 I took part into a Russian conference exclusively devoted to programming languages: <a href="http://plc.sfedu.ru/">Programming Languages and Compilers</a> (<a href="https://translate.google.com/translate?sl=auto&amp;tl=en&amp;js=y&amp;prev=_t&amp;hl=en&amp;ie=UTF-8&amp;u=http%3A%2F%2Fplc.sfedu.ru%2F&amp;edit-text=&amp;act=url">Google.Translated version of the site</a>). I was a member of organizing committee and had a paper there.</p> + +<p>This is the first conference in Russia highly focused on our area of PL. At least for the last several decades (I believe, there were conferences of the kind back in USSR). The conference was devoted to the memory of prominent Soviet PL-researcher from Rostov-on-Don, Adolf Fuksman who worked on ideas quite similar to what we know as the aspect-oriented programming back in the 70-s.</p> +<!-- more--> + +<p>We designed and implemented the conference with my colleagues from I.I.Vorovich institute of Mathematics, Mechanics and Computer Science, Southern Federal University (<a href="https://www.google.com/maps/place/Rostov-on-Don,+Rostov+Oblast,+Russia/@49.8345629,18.9321123,4.5z/data=!4m5!3m4!1s0x40e3c777c3b4b6ef:0x8248b451e48b4d04!8m2!3d47.2357137!4d39.701505">Rostov-on-Don</a>, Russia). We aimed at gathering as much PL-researchers and enthusiasts from academia in Russia as we could. One of the consequences of the aim was a decision to run the conference in Russian. Though we missed expertise from our non-Russian speaking colleagues, we got thorough participation from all over Russia:</p> + +<blockquote> + <p>Saint-Petersburg, Moscow, Novosibirsk, Krasnoyarsk, Ekaterinburg, Kazan, etc.</p></blockquote> + +<p>I only mention here the cities with more than 1 mil. population in decreasing in number of conference participants order (and excluding Rostov itself, of course).</p> + +<p>I particularly liked talks by invited speakers. When searching for ones, we targeted Russians who work at prominent universities and/or have some visibility at the international level. We ended up with two researchers: <a href="http://ilyasergey.net/">Ilya Sergey</a> (University College of London) and <a href="http://www.macs.hw.ac.uk/~ek19/">Ekaterina Komendantskaya</a> (Heriot-Watt U., Edinburg, UK). Interestingly, their talks were quite close to each other:</p> + +<ul> + <li>I. Sergey, Dependent Types for verification of real-world programs,</li> + <li>E. Komendantskaya, Automated Theorem Proving for Type Inference, Constructively.</li></ul> + +<p>Both of them talked about types as a logic tool to ensure program correctness.</p> + +<p>Biggest opening in this conference for me was a team from Languages Tools Laboratory of <a href="https://en.wikipedia.org/wiki/JetBrains">JetBrains</a>. Surely, you heard about JB: either about their famous IDE, IntelliJ IDEA, or the Kotlin programming language (which, by the way, <a href="https://blog.jetbrains.com/kotlin/2017/05/kotlin-on-android-now-official/">is endorsed</a> for Android development these days). You may also have noticed that JB become sponsors of ECOOP and OPLSS this year. So we had a whole team of researchers from Saint-Petersburg office of JB. Among their topics: <code>OCaml</code> embedding of <code>miniKanren</code> (some results was presented on ML Workshop 2016), parser combinator libraries for <code>OCaml</code> and constrained graph querying (this is not specifically a PL problem, see <a href="https://arxiv.org/abs/1502.02242">arXiv:1502.02242</a> for details).</p> + +<p>Otherwise the spectrum of topics presented on the conference was quite broad, here are some:</p> + +<ul> + <li>Static analysis with PVS-studio (a sponsor talk),</li> + <li>supercompilation (a talk by researchers from Pereslavl-Zalesskiy, where the topic is actively developed for decades),</li> + <li>C++ and beyond (by a member of the ISO C++ committee),</li> + <li>architecture-agnostic parallel programming languages and compilation techniques for parallel architectures,</li> + <li>game semantics and ontologies for PL semantics,</li> + <li>program analysis,</li> + <li>compiler architectures.</li></ul> + +<p>Full program with links to slides in Russian is available <a href="https://docs.google.com/spreadsheets/d/11QiFUqJG_NiBHVUfji_6-FiqP3aQWmdDBN13abM32nY/edit?usp=sharing">here</a>.</p> + +<p>Let me mention my submission: that was a joint work with a student of mine on exploring design space for parser combinator libraries using programming language with direct support of effect system, namely <a href="http://popl17.sigplan.org/event/popl-2017-papers-do-be-do-be-do">Frank</a>. <a href="http://staff.mmcs.sfedu.ru/%7Ejuliet/index.en.html">Julia Belyakova</a> also participated in the conference with her work on Coq-certified interpreter for an extension of lambda-calculus with concept-parameters (module-like kind of thing). The follow-up of that work is accepted for FTfJP workshop this year. You can also listen to her on the topic at the <a href="http://www.nepls.org/Events/30/">NEPLS</a> next week.</p> + +<p>I hope that we will find sources, time, and, most important, high quality submissions for PLC&ndash;2018.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Asumu-Takikawa.atom.xml b/blog/feeds/Author-Asumu-Takikawa.atom.xml new file mode 100644 index 00000000..dc89e93e --- /dev/null +++ b/blog/feeds/Author-Asumu-Takikawa.atom.xml @@ -0,0 +1,2280 @@ + + + PRL Blog: Posts tagged 'Author: Asumu Takikawa' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Asumu-Takikawa-html + 2016-07-11T17:33:40Z + + Tutorial: Racket FFI, part 3 + + urn:http-prl-ccs-neu-edu:-blog-2016-07-11-tutorial-racket-ffi-part-3 + 2016-07-11T17:33:40Z + 2016-07-11T17:33:40Z + + Asumu Takikawa + +<p>This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a> +and part 2 +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a>.</p> + +<p>In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.</p> +<!--more--> + +<p>As in the second post, let&rsquo;s start with some prologue code that establishes +the definitions from the previous two posts. But first, I&rsquo;m getting tired of +writing the <span class="stt">#:c-id identifier</span> notation for the underscored C function +names.</p> + +<p>Instead, let&rsquo;s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">$</span><span class="hspace">&nbsp;</span><span class="RktMeta">raco</span><span class="hspace">&nbsp;</span><span class="RktMeta">pkg</span><span class="hspace">&nbsp;</span><span class="RktMeta">install</span><span class="hspace">&nbsp;</span><span class="RktMeta">ffi-definer-convention</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>or you can just execute the following snippet in Racket:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pkg</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">pkg-install-command</span><span class="hspace">&nbsp;</span><span class="RktPn">#:skip-installed</span><span class="hspace">&nbsp;</span><span class="RktVal">#t</span><span class="hspace">&nbsp;</span><span class="RktVal">"ffi-definer-convention"</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Downloading repository git://github.com/takikawa/racket-ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: version: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: platform: x86_64-linux [3m]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: target machine: racket</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: installation name: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: variants: 3m</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main collects: /usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: collects paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main pkgs: /usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: pkgs paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: links files: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main docs: /usr/share/racket/doc</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- updating info-domain tables ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- pre-installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing foreign libraries --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing shared files ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- compiling collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- parallel build using 4 jobs ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 making: &lt;pkgs&gt;/ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- creating launchers --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing man pages --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- building documentation --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 running: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 rendering: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 2 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/local-redirect.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 1 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/release.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/search.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/start.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- post-installing collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This will install the package and compile its contents. If you&rsquo;re curious, the +docs for the package are available +<a href="http://docs.racket-lang.org/ffi-definer-convention/index.html">here</a>.</p> + +<p><span style="font-weight: bold">Note:</span> if you&rsquo;ve never installed a package before, you may want to glance +at the +<a href="http://docs.racket-lang.org/pkg/getting-started.html">package system docs</a>. +A tl;dr of packages is that they bundle Racket <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/collects.html#%28tech._collection%29"><span class="techinside">collections</span></a>, which are +sets of modules that you can refer to in a location-independent fashion such as +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> or <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28mod-path._racket%2Flist%29"><span class="RktSym">racket/list</span></a></span>.</p> + +<p>Anyhow, here&rsquo;s the prologue code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">avoid conflict with below</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._except-in%29%29">except-in</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">define-ffi-definer</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the new 3rd-party pkg</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi-definer-convention</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">describes how to transform from</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Racket to C ids</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:make-c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">convention:hyphen-&gt;underscore</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">note lack of #:c-id keyword arguments</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">(_cairo_t -&gt; Void) -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">do some drawing and give us the pict</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">f</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">f</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Notice that the <span class="RktWrap"><span class="RktSym">define-cairo</span></span> forms don&rsquo;t have any <span class="stt">#:c-id</span> keywords +anymore. Instead, the prologue code uses an overriden <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> +from my package that supports a <span class="stt">#:make-c-id</span> keyword that lets you specify +a naming convention to follow.</p> + +<p>Also, instead of creating a single bitmap and drawing into it, we now have a +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> function that takes a drawing function. When called, +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> will call the given function with a new bitmap object and +return the result.</p> + +<p>Now let&rsquo;s get to the main point of this blog post. Let&rsquo;s say that we want to play +with Cairo <a href="https://www.cairographics.org/manual/cairo-Paths.html">path</a> +objects this time. A path is +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-t">defined</a> +as a struct with the following structure:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_status_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">status</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it&rsquo;s +useful to define C types for the types of values in the path struct&rsquo;s fields. First, +let&rsquo;s specify that a <span class="stt">cairo_status_t</span> is an integer type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>It&rsquo;s actually an enum, but for the examples in this post we don&rsquo;t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-data-t">path data objects</a>. +Each path data object is a <span class="stt">cairo_path_data_t</span>, +which is specified with a C union:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">union</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">_cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_type_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">type</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">length</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">header</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">point</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Helpfully, the FFI library comes with support for unions with the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span></span> type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It&rsquo;s pretty +straightforward to specify this type too:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the path data type is just an enum</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">move-to</span><span class="hspace">&nbsp;</span><span class="RktVal">line-to</span><span class="hspace">&nbsp;</span><span class="RktVal">curve-to</span><span class="hspace">&nbsp;</span><span class="RktVal">close-path</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the header case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the point case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>There&rsquo;s a new type constructor here so let me explain that first. +The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span></span> constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span>, this constructor doesn&rsquo;t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.</p> + +<p>Each of the path data structs in the path data array will be manipulated with +the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">create a union from a list of doubles</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1.3</span><span class="hspace">&nbsp;</span><span class="RktVal">5.8</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">source type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">target type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;union&gt;</span></p></td></tr></tbody></table></div> + +<p>This snippet first construct a union object (via the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> +type) using a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span></span>. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it&rsquo;s an easy way +to generate a union object.</p> + +<p>The second line shows that a union prints as an opaque object. You can&rsquo;t do +anything with a union in Racket unless you project it to one of the sub-cases with +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></span> function. +This projection is <span class="emph">unsafe</span>, in the sense that if you don&rsquo;t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.</p> + +<p>More concretely, let&rsquo;s see what happens if we try to extract a value out of the +union both correctly and incorrectly:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">correct (matches construction)</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">cases are zero-indexed and ordered as written</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(1.3 5.8)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">incorrect, error</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">enum:int-&gt;_cairo_path_data_type_t: expected a known</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">#&lt;ctype:ufixint&gt;, got: 3435973837</span></p></td></tr></tbody></table></div> + +<p>Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.</p> + +<p>With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.</p> + +<p>With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it&rsquo;s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.</p> + +<p>So before moving on, let&rsquo;s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we&rsquo;ll figure out how to +deal with the array itself.</p> + +<h1><a name="(part._.Some_low-level_operations)"></a>Some low-level operations</h1> + +<p>Since we still don&rsquo;t have a C type for <span class="stt">cairo_path_t</span>, let&rsquo;s go ahead +and make a simple one where we punt on the work of specifying the array +type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In this type, we have specified the array as a bare <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span>. +For some added safety, we could also use something like +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__cpointer%29%29">_cpointer</a></span><span class="stt"> </span><span class="RktVal">'</span><span class="RktVal">cairo_status_t</span><span class="RktPn">)</span></span>, which sets up a tagged pointer +type like we saw in the first blog post with +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>.</p> + +<p>We&rsquo;ve seen the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type before, but haven&rsquo;t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.</p> + +<p>Before we get to that, let&rsquo;s go ahead and set up an FFI binding for +<span class="stt">cairo_copy_path</span> so that we can obtain a path struct to manipulate:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Do stuff to make the current</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">path non-empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Get the current path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/set_.html#%28form._%28%28quote._~23~25kernel%29._set%21%29%29">set!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Stroke clears the path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">so do it last</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<p>Note that <span class="RktWrap"><span class="RktSym">cairo-copy-path</span></span> gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span>, which +lets you dereference a pointer and access it at some concrete C type.</p> + +<p><span style="font-weight: bold">Note:</span> the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> function also takes an optional +offset argument which we will be used in an example later.</p> + +<p>For example, we can use <span class="RktWrap"><span class="RktSym">a-path</span></span> as a <span class="RktWrap"><span class="RktSym">_simple_cairo_path_t</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(0 #&lt;cpointer&gt; 8)</span></p></td></tr></tbody></table></div> + +<p>And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> again with an array type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">array</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">length field</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">array</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(#&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt;)</span></p></td></tr></tbody></table></div> + +<p>The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(move-to 2)</span></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(50.0 50.0)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">nonsense data here, wrong union case</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(4.2439915824246e-314 0.0)</span></p></td></tr></tbody></table></div> + +<p>One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.</p> + +<p>An alternative is to define a <span class="emph">custom C type</span> that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don&rsquo;t need to think about applying helper functions and +dereferencing pointers.</p> + +<h1><a name="(part._.Custom_.C_types)"></a>Custom C types</h1> + +<p>I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.</p> + +<p>In this way, it&rsquo;s possible to conduct interesting conversions, such as +dereferencing union objects automatically.</p> + +<p>Now let&rsquo;s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.</p> + +<p>First, we&rsquo;ll start by defining a struct type for the Racket representation +of Cairo paths:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define-struct.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._struct%29%29">struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ptr</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:property</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._prop~3asequence%29%29">prop:sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The representation will store one field <span class="RktWrap"><span class="RktSym">ptr</span></span> which, as the name +suggests, will store a pointer value. We&rsquo;ll see what to do with this +pointer later.</p> + +<p>This definition uses a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/structprops.html#%28tech._structure._type._property%29"><span class="techinside">structure type property</span></a> to make instances of +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> automatically work as sequences. This means that you +can iterate over them with a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span></span> loop or apply <span class="RktWrap"><span class="RktSym">sequence-ref</span></span> +on them. The property takes a function that takes an instance of the struct +type itself (here <span class="RktWrap"><span class="RktSym">p</span></span>) and that returns a sequence.</p> + +<p>We&rsquo;ll later define the <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span> function that will actually +construct the relevant sequence for us. For now, let&rsquo;s see how to construct +the C type given this struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract pointer out of representation</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">racket-&gt;c</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Just apply the Racket constructor</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">c-&gt;racket</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">racket-&gt;c</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">c-&gt;racket</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The base type for this <span class="RktWrap"><span class="RktSym">_cairo_path_t</span></span> is a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type. Since +the Cairo API returns pointers to new path values, it&rsquo;s hard to avoid using some +kind of pointer type as the base type here.</p> + +<p>This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we&rsquo;ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.</p> + +<p>The real work is done by the helper function that makes a +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> instance work as a sequence.</p> + +<p>Starting top-down, let&rsquo;s look at the definition of <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Cairo-Path -&gt; Sequence</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match-define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/values.html#%28def._%28%28quote._~23~25kernel%29._values%29%29">values</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The first thing the function does is extract the pointer out of the +representation, and then immediately calls <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.</p> + +<p><span style="font-weight: bold">Note:</span> in case you&rsquo;re not very familiar with Racket pattern matching, the +<span class="RktWrap"><span class="RktSym">match-define</span></span> form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml&rsquo;s <span class="stt">let</span> statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +<a href="http://docs.racket-lang.org/reference/match.html#%28form._%28%28lib._racket%2Fmatch..rkt%29._match-define%29%29">docs</a>.</p> + +<p>After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> function. Essentially, <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.</p> + +<p><span style="font-weight: bold">Note:</span> technically <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it&rsquo;s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.</p> + +<p>In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the <span class="RktWrap"><span class="RktSym">pos-&gt;element</span></span> function +and its helpers:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer -&gt; Integer -&gt; Element</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract the data path header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Length includes header, so subtract 1</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._sub1%29%29">sub1</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._add1%29%29">add1</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer Integer Integer -&gt; (Listof Data)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">offset argument</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in <span class="RktWrap"><span class="RktSym">get-points</span></span> +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.</p> + +<p>Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Note that determining the end of the sequence is very easy. It&rsquo;s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="stt"> </span><span class="RktSym">pos</span><span class="stt"> </span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></span>.</p> + +<p>Now we can try using a path as a sequence:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Using path as a sequence</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">elem</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="hspace">&nbsp;</span><span class="RktSym">elem</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 50.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (206.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (115.0 115.0))</span></p></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span></span> type.</p> + +<p>That concludes part 3 of the FFI tutorial. Hopefully you&rsquo;re now equipped +to deal with union types and custom C types. If not, see the +<a href="http://docs.racket-lang.org/foreign/index.html">FFI reference</a> +for more details on +<a href="http://docs.racket-lang.org/foreign/C_Union_Types.html">unions</a> +and +<a href="http://docs.racket-lang.org/foreign/ctype.html">custom C types</a>.</p> + +<p><span class="emph">Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!</span></p> + + Tutorial: Racket FFI, Part 2 + + urn:http-prl-ccs-neu-edu:-blog-2016-06-29-tutorial-racket-ffi-part-2 + 2016-06-29T18:48:17Z + 2016-06-29T18:48:17Z + + Asumu Takikawa + +<p>This is part 2 of my tutorial on using the Racket FFI. If you haven&rsquo;t read +part 1 yet, you can find it +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a>. +<span style="font-weight: bold">Update:</span> part 3 is also now available +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.</p> +<!--more--> + +<p>First, here&rsquo;s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">bitmap magic</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_create</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Bitmap -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a helper for displaying the bitmap</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<h1><a name="(part._.Dashes_and_array_arguments)"></a>Dashes and array arguments</h1> + +<p>To start off, let&rsquo;s look at another C example from the Cairo +<a href="https://www.cairographics.org/samples/">samples page</a>. +This time we will look at the "dash" example, which has a +use of an input array:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">sizeof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">)</span><span class="RktMeta">/sizeof</span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktVal">0</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">50.0</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">25.6</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_rel_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">102.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_curve_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The most interesting function here is <span class="stt">cairo_set_dash</span>, which takes an +array argument. The only other new functions are <span class="stt">cairo_rel_line_to</span> +and <span class="stt">cairo_curve_to</span> which have very straightforward C types:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-rel-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_rel_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-curve-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_curve_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Meanwhile, the C type signature for <span class="stt">cairo_set_dash</span> from the Cairo +docs looks like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Something to note about the arguments is that <span class="stt">num_dashes</span> +encodes the length of the array <span class="stt">dashes</span>. This will come up later when +we want to make the C type for this function more convenient.</p> + +<p>On the Racket side, it&rsquo;s natural to represent the array of dashes as either a +list or <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/vectors.html#%28tech._vector%29"><span class="techinside">vector</span></a> of numbers. Given that, a fairly literal translation of +the C type above might look like the following:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This type includes a type constructor we haven&rsquo;t seen yet: <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span>. +This is a so-called <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28tech._custom._function._type%29"><span class="techinside">custom function type</span></a> that has special meaning +inside of a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.</p> + +<p>Since we only want to provide a list from the Racket side to C, we&rsquo;ll use +the <span class="RktWrap"><span class="RktSym">i</span></span> input mode. We can then call the function like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">4</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Note that because of how we defined the type of <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it&rsquo;s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.</p> + +<p>Luckily, the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type constructor actually lets you do this with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">name</span><span class="stt"> </span><span class="RktSym">:</span><span class="stt"> </span><span class="RktSym">type</span><span class="RktPn">)</span></span> syntax for naming arguments in combination with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">type</span><span class="stt"> </span><span class="RktVar">=</span><span class="stt"> </span><span class="RktSym">expr</span><span class="RktPn">)</span></span> syntax for supplying computed arguments:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">name this argument for later uses in the type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a computed argument position</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3d%29%29">=</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>When a computed argument is specified with a <span class="RktWrap"><span class="RktVar">=</span></span>, it&rsquo;s not necessary to provide +the argument on the Racket side. So <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> is now an arity 3 +function that can be called like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This means we&rsquo;ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it&rsquo;s also possible to use Racket vectors instead of lists by using +the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__vector%29%29">_vector</a></span></span> type constructor.</p> + +<p>Putting it all together, we can reproduce the dashes example like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">25.6</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-rel-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-1</span>02.4</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-curve-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Result_arguments_and_.C_structs)"></a>Result arguments and C structs</h1> + +<p>For some more advanced FFI hacking, let&rsquo;s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Our challenge is to make a Racket function that takes a string (let&rsquo;s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let&rsquo;s just scale the +text to fit the width and assume the height will be okay.</p> + +<p>To implement the key step of measuring the text size, we can use the +<a href="https://www.cairographics.org/manual/cairo-text.html#cairo-text-extents"><span class="stt">cairo_text_extents</span></a> +function. Its type signature is as follows:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">char</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*utf8,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*extents</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The interesting part of this signature is that +<a href="https://www.cairographics.org/manual/cairo-cairo-scaled-font-t.html#cairo-text-extents-t"><span class="stt">cairo_text_extents_t</span></a> +is a struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">Cairo</span><span class="hspace">&nbsp;</span><span class="RktCmt">docs</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">height</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>We haven&rsquo;t yet seen how to handle C structs with the FFI, but it&rsquo;s not +too tricky. Support for C structs comes built-in and will look familiar +if you&rsquo;re used to Racket structs. We can directly translate the documented +definition above into a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span> declaration:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the leading underscore is mandatory</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">width</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">height</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">x-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">pointer to struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">allows NULL pointer</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer/null</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr></tbody></table></div> + +<p>Along with functions that look like regular Racket struct operations:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a struct constructor</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:make-cairo_text_extents_t&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field selector</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:cairo_text_extents_t-width&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a predicate for the struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t?</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:^TYPE?&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field mutation function</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">set-cairo_text_extents_t-width!</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:set-cairo_text_extents_t-width!&gt;</span></p></td></tr></tbody></table></div> + +<p>With the struct type defined, it&rsquo;s easy to come up with a rudimentary +interface for <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>This style of programming feels awfully imperative though. Since we&rsquo;re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span> FFI wrapper +by combining named arguments, a new <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> type constructor, +and a neat feature of <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> that lets you customize +the return result:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">named args and _ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">ext</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the return result of the C function</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">custom return result for the wrapper</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">ext</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> constructor works like the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span> constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the <span class="RktWrap"><span class="RktSym">o</span></span> mode to <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span>. +In output mode, this type will automatically allocate a new instance of the type +(using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._malloc%29%29">malloc</a></span></span> function) and arrange for it to be passed in as +a pointer.</p> + +<p>The strangest part of this example is that there are now two uses of the +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span></span> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.</p> + +<p>Using this new version of the wrapper is much simpler:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>With that in hand, it&rsquo;s pretty easy to write the function we set out to write:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-show-text</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_show_text</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-scale</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">String -&gt; Void</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">draws a string scaled horizontally</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">20</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">2.0</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._-%29%29"><span class="nobreak">-</span></a></span><span class="hspace">&nbsp;</span><span class="RktVal">256.0</span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym">width</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-scale</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-show-text</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>And to conclude part 2 of this tutorial, here&rsquo;s an example use +of the new <span class="RktWrap"><span class="RktSym">fit-text</span></span> function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktVal">"Saluton, Mondo / Hallo, mundo"</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + + Tutorial: Using Racket's FFI + + urn:http-prl-ccs-neu-edu:-blog-2016-06-27-tutorial-using-racket-s-ffi + 2016-06-27T16:22:11Z + 2016-06-27T16:22:11Z + + Asumu Takikawa + +<p><span style="font-weight: bold">Update:</span> this post is now part of a series. Part 2 is +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a> +and part 3 is +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>I&rsquo;ve seen several people ask for a tutorial on Racket&rsquo;s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +<a href="http://docs.racket-lang.org/foreign/index.html">documentation</a> +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.</p> + +<p>With that in mind, this blog post will provide a step-by-step tutorial +for Racket&rsquo;s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.</p> +<!--more--> + +<p>Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop <a href="http://www.ccs.neu.edu/racket/pubs/scheme04-bo.pdf">paper</a> +that you can read if you&rsquo;re curious about the design.</p> + +<p>The tutorial will focus on using the <a href="https://www.cairographics.org/">Cairo</a> +graphics library, mainly because it comes bundled with Racket.</p> + +<p>To start, let&rsquo;s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo&rsquo;s +<a href="https://www.cairographics.org/samples/">samples page</a>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">30.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">CAIRO_LINE_CAP_ROUND</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This uses the Racket drawing library <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/draw/index.html"><span class="RktSym">racket/draw</span></a></span> to construct +a bitmap object that we&rsquo;ll draw on. The <span class="RktWrap"><span class="RktSym">get-handle</span></span> method just extracts a +low-level Cairo surface value that we can use.</p> + +<p>NB: these code snippets don&rsquo;t come with a <span class="stt">#lang</span> declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy &amp; paste the snippets into your REPL.</p> + +<p>Our first real step is to import the FFI itself:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>As the module name suggests, the FFI is <span class="emph">unsafe</span> and can cause your Racket process +to segfault. If you&rsquo;re following along in DrRacket, you will want to save your file +frequently.</p> + +<p>Next, we can load the Cairo library to obtain a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28tech._foreign._library._value%29"><span class="techinside">foreign-library value</span></a>, which +is a handle that we use to access C values and functions:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Since Cairo has already been loaded by the Racket process because of the +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/gui/index.html"><span class="RktSym">racket/gui</span></a></span> import earlier, we can supply <span class="RktWrap"><span class="RktVal">#f</span></span> here as an +argument to <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span>. Normally you supply the name of a shared +library file such as <span class="RktWrap"><span class="RktVal">"libcairo"</span></span>:</p> + +<div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"libcairo"</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"2"</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></p></div> + +<p>The last list argument specifies the accepted versions (<span class="RktWrap"><span class="RktVal">#f</span></span> allows +a version-less library). For this post, those details aren&rsquo;t important but +see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span> if you&rsquo;re curious.</p> + +<h1><a name="(part._.Extracting_functions)"></a>Extracting functions</h1> + +<p>Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> function. The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> +function takes three arguments:</p> + +<ul> + <li> + <p>The name of the value as a string (or symbol or bytestring)</p></li> + <li> + <p>a foreign library value, and</p></li> + <li> + <p>a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/types.html#%28tech._c._type%29"><span class="techinside">C type</span></a>, which is a type description that tells +the FFI how to marshall between Racket and C.</p></li></ul> + +<p>C types are a crucial concept for the FFI. They range from relatively +simple types like <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> to more complicated +type constructors such as <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span>. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span></span> with two functions that +handle marshalling between C and Racket code.</p> + +<p>To make progress with our Cairo code, we need to create a drawing context from +the surface object <span class="RktWrap"><span class="RktSym">bt-surface</span></span> that we defined a while ago. The +relevant function in the Cairo docs is +<a href="https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-create"><span class="stt">cairo_create</span></a>, +which has the following type signature:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">NB:</span><span class="hspace">&nbsp;</span><span class="RktCmt">this</span><span class="hspace">&nbsp;</span><span class="RktCmt">is</span><span class="hspace">&nbsp;</span><span class="RktCmt">C</span><span class="hspace">&nbsp;</span><span class="RktCmt">code</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_create</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_surface_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*target</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a <span class="stt">cairo_surface_t</span> and +returns a pointer to a <span class="stt">cairo_t</span>. Let&rsquo;s start with a very simple C type +that matches up with this behavior: </div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span></p></div></div> + +<div class="SIntrapara">This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type.</div> + +<p>The following definition shows how to use this type to obtain a foreign +function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Then we can use <span class="RktWrap"><span class="RktSym">cairo-create</span></span> as an ordinary racket function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">ctx</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Interlude__more_type_safety)"></a>Interlude: more type safety</h1> + +<p>Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> types, it is easy +to accidentally misuse the function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">You may not want to actually run this</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a cairo_t is not a cairo_surface_t</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>To prevent such bad uses, it is good practice to use <span class="emph">tagged</span> pointer types +using <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>. Here are two example definitions that +correspond to the <span class="stt">cairo_t</span> and <span class="stt">cairo_surface_t</span> types from earlier:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">The leading underscores are mandatory</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>We can then redefine <span class="RktWrap"><span class="RktSym">cairo-create</span></span> with a better type, which will +prevent ill-typed calls:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">cairo_surface_t-&gt;C: argument is not non-null</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">`cairo_surface_t' pointer</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktErr">argument: #&lt;cpointer:cairo_t&gt;</span></p></td></tr></tbody></table></div> + +<p>Unfortunately our old definition of <span class="RktWrap"><span class="RktSym">ctx</span></span> doesn&rsquo;t have this tag:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#f</span></p></td></tr></tbody></table></div> + +<p>Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it&rsquo;s also possible to update +existing pointers with a tag like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-push-tag%21%29%29">cpointer-push-tag!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#t</span></p></td></tr></tbody></table></div> + +<p>Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).</p> + +<h1><a name="(part._.Macros_for_reducing_boilerplate)"></a>Macros for reducing boilerplate</h1> + +<p>Now let&rsquo;s start building the FFI bindings for the functions in the Cairo sample. +First, let&rsquo;s go ahead and look at all of the types for the sample functions +from the C API docs:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_cap_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">line_cap</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Starting with <span class="stt">cairo_move_to</span>, we can set up a definition like we did +with <span class="stt">cairo_create</span> before:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">"cairo_move_to"</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span> library that help reduce the +verbosity. Here&rsquo;s an alternative definition of <span class="RktWrap"><span class="RktSym">cairo-move-to</span></span> +using the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form from +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>As you can see, the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +<span class="RktWrap"><span class="RktSym">cairo_move_to</span></span>) you also don&rsquo;t need to supply the C name either.</p> + +<p>The definitions for <span class="stt">cairo_line_to</span>, <span class="stt">cairo_set_line_width</span>, and +<span class="stt">cairo_stroke</span> aren&rsquo;t very interesting, so I&rsquo;ll just include them +below without comment:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="stt">cairo_set_line_cap</span> case is more interesting because the type +<span class="stt">cairo_line_cap_t</span> is a C enumeration type. Racket&rsquo;s FFI comes with +convenience forms for defining enumeration types&#8212; + <wbr />though it&rsquo;s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here&rsquo;s a quote from the Barzilay and Orlovsky +paper on that:</p> + +<blockquote class="SubFlow"> + <p>Our design follows a simple principle: keep C-level +functionality to a minimum.</p></blockquote> + +<p>and specifically about enumerations:</p> + +<blockquote class="SubFlow"> + <p>For example, the C level part of our interface does not commit to a +specific implementation for enumerations &#8212; it simply exposes C integers.</p></blockquote> + +<p>To define an enumeration, we can use the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the <span class="stt">cairo_line_cap_t</span> type, it suffices to +just supply the cases as a list of symbols:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It&rsquo;s also possible to specify how the symbols map to integers more precisely +(see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> for those details).</p> + +<p>Given this type, we can specify the type for the line cap function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<h1><a name="(part._.Putting_it_all_together)"></a>Putting it all together</h1> + +<p>Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">30.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-cap</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">round</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object <span class="RktWrap"><span class="RktSym">bt</span></span>. To see the +results of what we drew, we can just evaluate <span class="RktWrap"><span class="RktSym">bt</span></span> at the REPL. But it&rsquo;s +a little nicer if we use the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> library to draw a frame around +it to distinguish it from the background:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>And we&rsquo;re done! Of course, there is a lot more to the FFI. For example, I haven&rsquo;t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I&rsquo;m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!</p> + + Gradual Typing Across the Spectrum + + urn:http-prl-ccs-neu-edu:-blog-2016-05-18-gradual-typing-across-the-spectrum + 2016-05-18T07:58:56Z + 2016-05-18T07:58:56Z + + Asumu Takikawa + +<blockquote> + <p>Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. &mdash; Matthias Felleisen</p></blockquote> + +<p>Yesterday we hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found <a href="http://prl.ccs.neu.edu/gtp/pi2016/pi2016.html">here</a>). In case you aren&rsquo;t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.</p> + +<p>Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing&mdash;the ideas that cross language and platform barriers.</p> +<!-- more--> + +<p>There were a good number of both institutions and programming languages represented at the meeting, with researchers from all of <a href="http://cs.brown.edu/research/plt/">Brown University</a>, <a href="https://wonks.github.io/">Indiana University</a>, <a href="http://prl.ccs.neu.edu/">Northeastern University</a>, and the <a href="http://www.cs.umd.edu/projects/PL/">University of Maryland</a>. The languages that we work on cover a broad subset of the dynamically-typed languages: Clojure, JavaScript, R, Racket, Ruby, Pyret, and Python.</p> + +<div class="figure"><img src="/img/2016-day-slide-4.png" alt="" /> + <p class="caption"></p></div> + +<p>The specific research artifacts that were represented include <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>, <a href="https://github.com/plum-umd/rdl">RDL</a> (contracts for Ruby), <a href="http://plg.uwaterloo.ca/~dynjs/strongscript/">StrongScript</a>, <a href="http://typedclojure.org/">Typed Clojure</a>, and <a href="http://docs.racket-lang.org/ts-guide/index.html">Typed Racket</a>.</p> + +<p>In this blog post, I&rsquo;ll summarize some of the key research themes that were brought up at the meeting. Since I can&rsquo;t go into too much detail about every topic, I will link to the relevant research papers and other resources.</p> + +<p>At a high level, the talks covered four major facets of gradual typing: expressiveness, performance, usability, and implementation techniques.</p> + +<h2 id="expressiveness">Expressiveness</h2> + +<p>By expressiveness, I mean what kinds of language features a gradual type system supports and the richness of the reasoning that the type system provides. Since gradual typing is about augmenting existing dynamically-typed languages, a gradual type system should support the language features that programmers actually use.</p> + +<p>This is why recent implementations of gradual typing have focused on enabling object-oriented programming, since objects are widely used in nearly all dynamically-typed languages in use today. Unfortunately, since different languages have wildly different object systems, it&rsquo;s hard to compare research on gradual OO languages. Ben Chung is working to address this by coming up with a formal model that tries to unify various accounts of objects in order to better explain the design tradeoffs. His goal is to cover the core ideas in Reticulated Python, StrongScript, and Typed Racket.</p> + +<p>Of course, dynamically-typed languages have a lot more than OO features. Along these lines, I gave a talk on how at NU we&rsquo;re working to extend Typed Racket to cover everything from objects (my thesis topic), first-class modules (Dan Feltey&rsquo;s MS project), and higher-order contracts (Brian LaChance&rsquo;s MS project).</p> + +<p>On the other side, as programs get more complex, programmers may wish to write richer type specifications that provide even more guarantees. This makes gradual typing a wide spectrum that goes from completely untyped, fully typed, and then beyond to dependently typed. Andrew Kent and David Christiansen both presented work that takes gradual typing beyond ordinary typed reasoning with dependent types.</p> + +<p>Andrew presented an extension of Typed Racket that adds type refinements that can check rich properties (like vector bounds) that are found in real Racket code (see his RacketCon <a href="https://www.youtube.com/watch?v=ejFJIAsvdEg">talk</a> and recent <a href="http://arxiv.org/pdf/1511.07033.pdf">PLDI paper</a>). David Christiansen followed with a talk about adding dependent type theory to Typed Racket, which would allow correct-by-construction programming using a Nuprl-like proof system (he had a very cool GUI proof assistant demo in his slides!).</p> + +<h2 id="performance">Performance</h2> + +<div class="figure"><img src="/img/2016-day-slide-8.png" alt="" /> + <p class="caption"></p></div> + +<p>One of the key practical concerns about gradual typing is its performance overhead. It&rsquo;s a concern because in order to ensure type safety, a gradually-typed language implementation needs to install dynamic checks between the typed and untyped parts of a program. This catches any inconsistencies between the typed interfaces and how the untyped code may call into them.</p> + +<p>Ben Greenman gave an upbeat talk that set the stage for this topic, pointing out some key lessons that we&rsquo;ve learned about performance from building Typed Racket. The main idea he presented (also the topic of our <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016 paper</a>) is that to evaluate a gradual type system, you want to explore different ways of adding types to a program and see how much it costs. This evaluation effort started with Typed Racket, but he and Zeina Migeed are working on expanding it to Reticulated Python.</p> + +<p>From IU, Andre Kuhlenschmidt and Deyaaeldeen Almahallawi are exploring how ahead-of-time (AOT) compilation strategies could help reduce the cost of gradual typing. In particular, they are working on implementing <a href="https://github.com/deyaaeldeen/Schml">Schml</a>: a compiler from the gradually-typed lambda calculus to C.</p> + +<p>In addition to AOT compilation, the folks at IU are exploring tracing JIT compilation as a means to make gradual typing faster. More specifically, Spenser Bauman talked about Pycket, an alternative implementation of Racket that uses RPython/PyPy to dramatically lower the overhead of gradual typing (also see the <a href="https://www.youtube.com/watch?v=GOfIY8NHAqg">recording</a> of Spenser&rsquo;s talk on the topic at RacketCon and his <a href="http://homes.soic.indiana.edu/samth/pycket-draft.pdf">ICFP paper</a>).</p> + +<h2 id="usability">Usability</h2> + +<p>On the usability side, both Shriram Krishnamurthi and Ambrose Bonnaire-Sergeant made observations on what it takes to get gradual typing in the hands of real software developers.</p> + +<p>Shriram approached the topic from the angle of CS education, which is the focus of the <a href="http://www.pyret.org">Pyret</a> language, and shared what the Brown language group is working on. While Pyret doesn&rsquo;t exactly fit the mold of gradual typing, it&rsquo;s a close cousin since it&rsquo;s a dynamically-typed language that explicitly takes design cues from the best parts of statically-typed languages. That approach lets CS beginners think in terms of types (the approach spearheaded by <a href="http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html">HtDP</a> and <a href="http://www.bootstrapworld.org/">Bootstrap</a>) without having to battle a typechecker from the start.</p> + +<p>For professional software developers, a major concern with gradual typing is that writing type annotations may be a tedious and time intensive task. Ambrose, who is the creator of Typed Clojure, shared some preliminary work on how to cut down on the tedium by inferring gradual type annotations by instrumenting programs for a dynamic analysis. The hope is to be able to infer both recursive and polymorphic type annotations automatically from tests (you may also be interested in Ambrose&rsquo;s recent <a href="http://frenchy64.github.io/papers/esop16-short.pdf">ESOP paper</a> on Typed Clojure).</p> + +<h2 id="implementation-techniques">Implementation Techniques</h2> + +<p>Finally, several talks focused on alternative implementation techniques for gradual typing that provide a variety of software engineering benefits for implementers.</p> + +<p>From Maryland, Brianna Ren gave a talk on Hummingbird, a just-in-time typechecker for Ruby programs (also see the upcoming <a href="http://www.cs.umd.edu/~jfoster/papers/pldi16.pdf">PLDI paper</a> by Brianna and Jeff Foster). The basic idea is that it&rsquo;s hard to implement a traditional static typechecker for a language that heavily relies on metaprogramming, in which the fields/methods of classes may be rewritten at run-time. This is particularly tricky for frameworks like Ruby on Rails. Instead of checking types at compile-time, Hummingbird actually executes the typechecker at run-time in order to be able to accurately check programs that use run-time metaprogramming. To reduce overheads, she uses a cache for typechecking that is invalidated when classes are modified.</p> + +<p>Stephen Chang gave a very different view on metaprogramming in his talk, which focused on <em>implementing</em> typecheckers using metaprogramming (the <a href="http://docs.racket-lang.org/trivial/index.html">trivial</a> Typed Racket library is an offshoot of this work). His key idea is that typecheckers share many aspects with macro-based metaprogramming systems, such as the need to traverse syntax and annotate it with information. Since they share so much in common, why not just implement the typechecker as a macro? Stephen demonstrates that not only is this possible, but that it&rsquo;s possible to implement a wide variety of type system features this way including (local) type inference. The connection to gradual typing is that even a gradual type system can be implemented as a metaprogram by integrating the generation of dynamic checks into the macro transformation process.</p> + +<p>The last talk of the day (but certainly not the least), was by Michael Vitousek, who focused on the <em>transient</em> implementation of gradual typing (first described in his <a href="http://homes.soic.indiana.edu/mvitouse/papers/dls14.pdf">DLS paper</a>). Traditionally, gradual type systems have implemented their dynamic checks using <a href="https://en.wikipedia.org/wiki/Proxy_pattern">proxy</a> objects that wrap method implementations with both pre- and post-checks. Unfortunately, this implementation technique often conflicts with the underlying language. Since proxying changes the identity of an object it can interfere with object equality tests. Instead, the transient approach bakes the dynamic checks into and throughout the typed code to implement a &ldquo;defense in depth&rdquo; against inconsistencies with untyped code. The great thing about this implementation technique is that it doesn&rsquo;t demand any specialized support from the underlying language runtime and is therefore easy to port to other languages (like JavaScript).</p> + +<h2 id="conclusion">Conclusion</h2> + +<div class="figure"><img src="/img/2016-day-slide-3.png" alt="" /> + <p class="caption"></p></div> + +<p>Hopefully this blog post helps provide a better picture of the state of gradual typing research. The exciting thing about gradual typing is that it contains both interesting theoretical problems and also connects to the practical needs of software developers.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Asumu-Takikawa.rss.xml b/blog/feeds/Author-Asumu-Takikawa.rss.xml new file mode 100644 index 00000000..d7d87808 --- /dev/null +++ b/blog/feeds/Author-Asumu-Takikawa.rss.xml @@ -0,0 +1,2274 @@ + + + + PRL Blog: Posts tagged 'Author: Asumu Takikawa' + PRL Blog: Posts tagged 'Author: Asumu Takikawa' + http://prl.ccs.neu.edu/blog/tags/Author-Asumu-Takikawa.html + Mon, 11 Jul 2016 17:33:40 UT + Mon, 11 Jul 2016 17:33:40 UT + 1800 + + Tutorial: Racket FFI, part 3 + http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/?utm_source=Author-Asumu-Takikawa&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-07-11-tutorial-racket-ffi-part-3 + Mon, 11 Jul 2016 17:33:40 UT + Asumu Takikawa + +<p>This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a> +and part 2 +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a>.</p> + +<p>In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.</p> +<!--more--> + +<p>As in the second post, let&rsquo;s start with some prologue code that establishes +the definitions from the previous two posts. But first, I&rsquo;m getting tired of +writing the <span class="stt">#:c-id identifier</span> notation for the underscored C function +names.</p> + +<p>Instead, let&rsquo;s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">$</span><span class="hspace">&nbsp;</span><span class="RktMeta">raco</span><span class="hspace">&nbsp;</span><span class="RktMeta">pkg</span><span class="hspace">&nbsp;</span><span class="RktMeta">install</span><span class="hspace">&nbsp;</span><span class="RktMeta">ffi-definer-convention</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>or you can just execute the following snippet in Racket:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pkg</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">pkg-install-command</span><span class="hspace">&nbsp;</span><span class="RktPn">#:skip-installed</span><span class="hspace">&nbsp;</span><span class="RktVal">#t</span><span class="hspace">&nbsp;</span><span class="RktVal">"ffi-definer-convention"</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Downloading repository git://github.com/takikawa/racket-ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: version: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: platform: x86_64-linux [3m]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: target machine: racket</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: installation name: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: variants: 3m</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main collects: /usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: collects paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main pkgs: /usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: pkgs paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: links files: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main docs: /usr/share/racket/doc</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- updating info-domain tables ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- pre-installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing foreign libraries --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing shared files ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- compiling collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- parallel build using 4 jobs ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 making: &lt;pkgs&gt;/ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- creating launchers --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing man pages --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- building documentation --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 running: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 rendering: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 2 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/local-redirect.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 1 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/release.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/search.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/start.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- post-installing collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This will install the package and compile its contents. If you&rsquo;re curious, the +docs for the package are available +<a href="http://docs.racket-lang.org/ffi-definer-convention/index.html">here</a>.</p> + +<p><span style="font-weight: bold">Note:</span> if you&rsquo;ve never installed a package before, you may want to glance +at the +<a href="http://docs.racket-lang.org/pkg/getting-started.html">package system docs</a>. +A tl;dr of packages is that they bundle Racket <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/collects.html#%28tech._collection%29"><span class="techinside">collections</span></a>, which are +sets of modules that you can refer to in a location-independent fashion such as +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> or <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28mod-path._racket%2Flist%29"><span class="RktSym">racket/list</span></a></span>.</p> + +<p>Anyhow, here&rsquo;s the prologue code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">avoid conflict with below</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._except-in%29%29">except-in</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">define-ffi-definer</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the new 3rd-party pkg</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi-definer-convention</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">describes how to transform from</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Racket to C ids</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:make-c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">convention:hyphen-&gt;underscore</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">note lack of #:c-id keyword arguments</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">(_cairo_t -&gt; Void) -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">do some drawing and give us the pict</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">f</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">f</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Notice that the <span class="RktWrap"><span class="RktSym">define-cairo</span></span> forms don&rsquo;t have any <span class="stt">#:c-id</span> keywords +anymore. Instead, the prologue code uses an overriden <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> +from my package that supports a <span class="stt">#:make-c-id</span> keyword that lets you specify +a naming convention to follow.</p> + +<p>Also, instead of creating a single bitmap and drawing into it, we now have a +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> function that takes a drawing function. When called, +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> will call the given function with a new bitmap object and +return the result.</p> + +<p>Now let&rsquo;s get to the main point of this blog post. Let&rsquo;s say that we want to play +with Cairo <a href="https://www.cairographics.org/manual/cairo-Paths.html">path</a> +objects this time. A path is +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-t">defined</a> +as a struct with the following structure:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_status_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">status</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it&rsquo;s +useful to define C types for the types of values in the path struct&rsquo;s fields. First, +let&rsquo;s specify that a <span class="stt">cairo_status_t</span> is an integer type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>It&rsquo;s actually an enum, but for the examples in this post we don&rsquo;t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-data-t">path data objects</a>. +Each path data object is a <span class="stt">cairo_path_data_t</span>, +which is specified with a C union:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">union</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">_cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_type_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">type</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">length</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">header</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">point</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Helpfully, the FFI library comes with support for unions with the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span></span> type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It&rsquo;s pretty +straightforward to specify this type too:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the path data type is just an enum</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">move-to</span><span class="hspace">&nbsp;</span><span class="RktVal">line-to</span><span class="hspace">&nbsp;</span><span class="RktVal">curve-to</span><span class="hspace">&nbsp;</span><span class="RktVal">close-path</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the header case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the point case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>There&rsquo;s a new type constructor here so let me explain that first. +The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span></span> constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span>, this constructor doesn&rsquo;t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.</p> + +<p>Each of the path data structs in the path data array will be manipulated with +the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">create a union from a list of doubles</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1.3</span><span class="hspace">&nbsp;</span><span class="RktVal">5.8</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">source type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">target type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;union&gt;</span></p></td></tr></tbody></table></div> + +<p>This snippet first construct a union object (via the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> +type) using a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span></span>. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it&rsquo;s an easy way +to generate a union object.</p> + +<p>The second line shows that a union prints as an opaque object. You can&rsquo;t do +anything with a union in Racket unless you project it to one of the sub-cases with +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></span> function. +This projection is <span class="emph">unsafe</span>, in the sense that if you don&rsquo;t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.</p> + +<p>More concretely, let&rsquo;s see what happens if we try to extract a value out of the +union both correctly and incorrectly:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">correct (matches construction)</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">cases are zero-indexed and ordered as written</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(1.3 5.8)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">incorrect, error</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">enum:int-&gt;_cairo_path_data_type_t: expected a known</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">#&lt;ctype:ufixint&gt;, got: 3435973837</span></p></td></tr></tbody></table></div> + +<p>Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.</p> + +<p>With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.</p> + +<p>With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it&rsquo;s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.</p> + +<p>So before moving on, let&rsquo;s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we&rsquo;ll figure out how to +deal with the array itself.</p> + +<h1><a name="(part._.Some_low-level_operations)"></a>Some low-level operations</h1> + +<p>Since we still don&rsquo;t have a C type for <span class="stt">cairo_path_t</span>, let&rsquo;s go ahead +and make a simple one where we punt on the work of specifying the array +type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In this type, we have specified the array as a bare <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span>. +For some added safety, we could also use something like +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__cpointer%29%29">_cpointer</a></span><span class="stt"> </span><span class="RktVal">'</span><span class="RktVal">cairo_status_t</span><span class="RktPn">)</span></span>, which sets up a tagged pointer +type like we saw in the first blog post with +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>.</p> + +<p>We&rsquo;ve seen the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type before, but haven&rsquo;t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.</p> + +<p>Before we get to that, let&rsquo;s go ahead and set up an FFI binding for +<span class="stt">cairo_copy_path</span> so that we can obtain a path struct to manipulate:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Do stuff to make the current</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">path non-empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Get the current path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/set_.html#%28form._%28%28quote._~23~25kernel%29._set%21%29%29">set!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Stroke clears the path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">so do it last</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<p>Note that <span class="RktWrap"><span class="RktSym">cairo-copy-path</span></span> gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span>, which +lets you dereference a pointer and access it at some concrete C type.</p> + +<p><span style="font-weight: bold">Note:</span> the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> function also takes an optional +offset argument which we will be used in an example later.</p> + +<p>For example, we can use <span class="RktWrap"><span class="RktSym">a-path</span></span> as a <span class="RktWrap"><span class="RktSym">_simple_cairo_path_t</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(0 #&lt;cpointer&gt; 8)</span></p></td></tr></tbody></table></div> + +<p>And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> again with an array type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">array</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">length field</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">array</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(#&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt;)</span></p></td></tr></tbody></table></div> + +<p>The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(move-to 2)</span></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(50.0 50.0)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">nonsense data here, wrong union case</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(4.2439915824246e-314 0.0)</span></p></td></tr></tbody></table></div> + +<p>One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.</p> + +<p>An alternative is to define a <span class="emph">custom C type</span> that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don&rsquo;t need to think about applying helper functions and +dereferencing pointers.</p> + +<h1><a name="(part._.Custom_.C_types)"></a>Custom C types</h1> + +<p>I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.</p> + +<p>In this way, it&rsquo;s possible to conduct interesting conversions, such as +dereferencing union objects automatically.</p> + +<p>Now let&rsquo;s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.</p> + +<p>First, we&rsquo;ll start by defining a struct type for the Racket representation +of Cairo paths:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define-struct.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._struct%29%29">struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ptr</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:property</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._prop~3asequence%29%29">prop:sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The representation will store one field <span class="RktWrap"><span class="RktSym">ptr</span></span> which, as the name +suggests, will store a pointer value. We&rsquo;ll see what to do with this +pointer later.</p> + +<p>This definition uses a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/structprops.html#%28tech._structure._type._property%29"><span class="techinside">structure type property</span></a> to make instances of +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> automatically work as sequences. This means that you +can iterate over them with a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span></span> loop or apply <span class="RktWrap"><span class="RktSym">sequence-ref</span></span> +on them. The property takes a function that takes an instance of the struct +type itself (here <span class="RktWrap"><span class="RktSym">p</span></span>) and that returns a sequence.</p> + +<p>We&rsquo;ll later define the <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span> function that will actually +construct the relevant sequence for us. For now, let&rsquo;s see how to construct +the C type given this struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract pointer out of representation</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">racket-&gt;c</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Just apply the Racket constructor</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">c-&gt;racket</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">racket-&gt;c</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">c-&gt;racket</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The base type for this <span class="RktWrap"><span class="RktSym">_cairo_path_t</span></span> is a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type. Since +the Cairo API returns pointers to new path values, it&rsquo;s hard to avoid using some +kind of pointer type as the base type here.</p> + +<p>This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we&rsquo;ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.</p> + +<p>The real work is done by the helper function that makes a +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> instance work as a sequence.</p> + +<p>Starting top-down, let&rsquo;s look at the definition of <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Cairo-Path -&gt; Sequence</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match-define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/values.html#%28def._%28%28quote._~23~25kernel%29._values%29%29">values</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The first thing the function does is extract the pointer out of the +representation, and then immediately calls <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.</p> + +<p><span style="font-weight: bold">Note:</span> in case you&rsquo;re not very familiar with Racket pattern matching, the +<span class="RktWrap"><span class="RktSym">match-define</span></span> form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml&rsquo;s <span class="stt">let</span> statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +<a href="http://docs.racket-lang.org/reference/match.html#%28form._%28%28lib._racket%2Fmatch..rkt%29._match-define%29%29">docs</a>.</p> + +<p>After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> function. Essentially, <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.</p> + +<p><span style="font-weight: bold">Note:</span> technically <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it&rsquo;s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.</p> + +<p>In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the <span class="RktWrap"><span class="RktSym">pos-&gt;element</span></span> function +and its helpers:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer -&gt; Integer -&gt; Element</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract the data path header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Length includes header, so subtract 1</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._sub1%29%29">sub1</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._add1%29%29">add1</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer Integer Integer -&gt; (Listof Data)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">offset argument</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in <span class="RktWrap"><span class="RktSym">get-points</span></span> +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.</p> + +<p>Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Note that determining the end of the sequence is very easy. It&rsquo;s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="stt"> </span><span class="RktSym">pos</span><span class="stt"> </span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></span>.</p> + +<p>Now we can try using a path as a sequence:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Using path as a sequence</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">elem</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="hspace">&nbsp;</span><span class="RktSym">elem</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 50.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (206.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (115.0 115.0))</span></p></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span></span> type.</p> + +<p>That concludes part 3 of the FFI tutorial. Hopefully you&rsquo;re now equipped +to deal with union types and custom C types. If not, see the +<a href="http://docs.racket-lang.org/foreign/index.html">FFI reference</a> +for more details on +<a href="http://docs.racket-lang.org/foreign/C_Union_Types.html">unions</a> +and +<a href="http://docs.racket-lang.org/foreign/ctype.html">custom C types</a>.</p> + +<p><span class="emph">Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!</span></p> + + Tutorial: Racket FFI, Part 2 + http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/?utm_source=Author-Asumu-Takikawa&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-29-tutorial-racket-ffi-part-2 + Wed, 29 Jun 2016 18:48:17 UT + Asumu Takikawa + +<p>This is part 2 of my tutorial on using the Racket FFI. If you haven&rsquo;t read +part 1 yet, you can find it +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a>. +<span style="font-weight: bold">Update:</span> part 3 is also now available +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.</p> +<!--more--> + +<p>First, here&rsquo;s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">bitmap magic</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_create</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Bitmap -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a helper for displaying the bitmap</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<h1><a name="(part._.Dashes_and_array_arguments)"></a>Dashes and array arguments</h1> + +<p>To start off, let&rsquo;s look at another C example from the Cairo +<a href="https://www.cairographics.org/samples/">samples page</a>. +This time we will look at the "dash" example, which has a +use of an input array:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">sizeof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">)</span><span class="RktMeta">/sizeof</span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktVal">0</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">50.0</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">25.6</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_rel_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">102.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_curve_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The most interesting function here is <span class="stt">cairo_set_dash</span>, which takes an +array argument. The only other new functions are <span class="stt">cairo_rel_line_to</span> +and <span class="stt">cairo_curve_to</span> which have very straightforward C types:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-rel-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_rel_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-curve-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_curve_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Meanwhile, the C type signature for <span class="stt">cairo_set_dash</span> from the Cairo +docs looks like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Something to note about the arguments is that <span class="stt">num_dashes</span> +encodes the length of the array <span class="stt">dashes</span>. This will come up later when +we want to make the C type for this function more convenient.</p> + +<p>On the Racket side, it&rsquo;s natural to represent the array of dashes as either a +list or <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/vectors.html#%28tech._vector%29"><span class="techinside">vector</span></a> of numbers. Given that, a fairly literal translation of +the C type above might look like the following:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This type includes a type constructor we haven&rsquo;t seen yet: <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span>. +This is a so-called <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28tech._custom._function._type%29"><span class="techinside">custom function type</span></a> that has special meaning +inside of a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.</p> + +<p>Since we only want to provide a list from the Racket side to C, we&rsquo;ll use +the <span class="RktWrap"><span class="RktSym">i</span></span> input mode. We can then call the function like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">4</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Note that because of how we defined the type of <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it&rsquo;s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.</p> + +<p>Luckily, the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type constructor actually lets you do this with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">name</span><span class="stt"> </span><span class="RktSym">:</span><span class="stt"> </span><span class="RktSym">type</span><span class="RktPn">)</span></span> syntax for naming arguments in combination with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">type</span><span class="stt"> </span><span class="RktVar">=</span><span class="stt"> </span><span class="RktSym">expr</span><span class="RktPn">)</span></span> syntax for supplying computed arguments:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">name this argument for later uses in the type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a computed argument position</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3d%29%29">=</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>When a computed argument is specified with a <span class="RktWrap"><span class="RktVar">=</span></span>, it&rsquo;s not necessary to provide +the argument on the Racket side. So <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> is now an arity 3 +function that can be called like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This means we&rsquo;ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it&rsquo;s also possible to use Racket vectors instead of lists by using +the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__vector%29%29">_vector</a></span></span> type constructor.</p> + +<p>Putting it all together, we can reproduce the dashes example like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">25.6</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-rel-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-1</span>02.4</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-curve-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Result_arguments_and_.C_structs)"></a>Result arguments and C structs</h1> + +<p>For some more advanced FFI hacking, let&rsquo;s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Our challenge is to make a Racket function that takes a string (let&rsquo;s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let&rsquo;s just scale the +text to fit the width and assume the height will be okay.</p> + +<p>To implement the key step of measuring the text size, we can use the +<a href="https://www.cairographics.org/manual/cairo-text.html#cairo-text-extents"><span class="stt">cairo_text_extents</span></a> +function. Its type signature is as follows:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">char</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*utf8,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*extents</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The interesting part of this signature is that +<a href="https://www.cairographics.org/manual/cairo-cairo-scaled-font-t.html#cairo-text-extents-t"><span class="stt">cairo_text_extents_t</span></a> +is a struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">Cairo</span><span class="hspace">&nbsp;</span><span class="RktCmt">docs</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">height</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>We haven&rsquo;t yet seen how to handle C structs with the FFI, but it&rsquo;s not +too tricky. Support for C structs comes built-in and will look familiar +if you&rsquo;re used to Racket structs. We can directly translate the documented +definition above into a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span> declaration:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the leading underscore is mandatory</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">width</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">height</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">x-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">pointer to struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">allows NULL pointer</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer/null</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr></tbody></table></div> + +<p>Along with functions that look like regular Racket struct operations:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a struct constructor</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:make-cairo_text_extents_t&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field selector</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:cairo_text_extents_t-width&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a predicate for the struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t?</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:^TYPE?&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field mutation function</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">set-cairo_text_extents_t-width!</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:set-cairo_text_extents_t-width!&gt;</span></p></td></tr></tbody></table></div> + +<p>With the struct type defined, it&rsquo;s easy to come up with a rudimentary +interface for <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>This style of programming feels awfully imperative though. Since we&rsquo;re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span> FFI wrapper +by combining named arguments, a new <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> type constructor, +and a neat feature of <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> that lets you customize +the return result:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">named args and _ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">ext</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the return result of the C function</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">custom return result for the wrapper</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">ext</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> constructor works like the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span> constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the <span class="RktWrap"><span class="RktSym">o</span></span> mode to <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span>. +In output mode, this type will automatically allocate a new instance of the type +(using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._malloc%29%29">malloc</a></span></span> function) and arrange for it to be passed in as +a pointer.</p> + +<p>The strangest part of this example is that there are now two uses of the +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span></span> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.</p> + +<p>Using this new version of the wrapper is much simpler:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>With that in hand, it&rsquo;s pretty easy to write the function we set out to write:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-show-text</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_show_text</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-scale</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">String -&gt; Void</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">draws a string scaled horizontally</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">20</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">2.0</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._-%29%29"><span class="nobreak">-</span></a></span><span class="hspace">&nbsp;</span><span class="RktVal">256.0</span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym">width</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-scale</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-show-text</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>And to conclude part 2 of this tutorial, here&rsquo;s an example use +of the new <span class="RktWrap"><span class="RktSym">fit-text</span></span> function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktVal">"Saluton, Mondo / Hallo, mundo"</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + + Tutorial: Using Racket's FFI + http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/?utm_source=Author-Asumu-Takikawa&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-27-tutorial-using-racket-s-ffi + Mon, 27 Jun 2016 16:22:11 UT + Asumu Takikawa + +<p><span style="font-weight: bold">Update:</span> this post is now part of a series. Part 2 is +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a> +and part 3 is +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>I&rsquo;ve seen several people ask for a tutorial on Racket&rsquo;s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +<a href="http://docs.racket-lang.org/foreign/index.html">documentation</a> +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.</p> + +<p>With that in mind, this blog post will provide a step-by-step tutorial +for Racket&rsquo;s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.</p> +<!--more--> + +<p>Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop <a href="http://www.ccs.neu.edu/racket/pubs/scheme04-bo.pdf">paper</a> +that you can read if you&rsquo;re curious about the design.</p> + +<p>The tutorial will focus on using the <a href="https://www.cairographics.org/">Cairo</a> +graphics library, mainly because it comes bundled with Racket.</p> + +<p>To start, let&rsquo;s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo&rsquo;s +<a href="https://www.cairographics.org/samples/">samples page</a>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">30.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">CAIRO_LINE_CAP_ROUND</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This uses the Racket drawing library <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/draw/index.html"><span class="RktSym">racket/draw</span></a></span> to construct +a bitmap object that we&rsquo;ll draw on. The <span class="RktWrap"><span class="RktSym">get-handle</span></span> method just extracts a +low-level Cairo surface value that we can use.</p> + +<p>NB: these code snippets don&rsquo;t come with a <span class="stt">#lang</span> declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy &amp; paste the snippets into your REPL.</p> + +<p>Our first real step is to import the FFI itself:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>As the module name suggests, the FFI is <span class="emph">unsafe</span> and can cause your Racket process +to segfault. If you&rsquo;re following along in DrRacket, you will want to save your file +frequently.</p> + +<p>Next, we can load the Cairo library to obtain a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28tech._foreign._library._value%29"><span class="techinside">foreign-library value</span></a>, which +is a handle that we use to access C values and functions:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Since Cairo has already been loaded by the Racket process because of the +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/gui/index.html"><span class="RktSym">racket/gui</span></a></span> import earlier, we can supply <span class="RktWrap"><span class="RktVal">#f</span></span> here as an +argument to <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span>. Normally you supply the name of a shared +library file such as <span class="RktWrap"><span class="RktVal">"libcairo"</span></span>:</p> + +<div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"libcairo"</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"2"</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></p></div> + +<p>The last list argument specifies the accepted versions (<span class="RktWrap"><span class="RktVal">#f</span></span> allows +a version-less library). For this post, those details aren&rsquo;t important but +see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span> if you&rsquo;re curious.</p> + +<h1><a name="(part._.Extracting_functions)"></a>Extracting functions</h1> + +<p>Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> function. The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> +function takes three arguments:</p> + +<ul> + <li> + <p>The name of the value as a string (or symbol or bytestring)</p></li> + <li> + <p>a foreign library value, and</p></li> + <li> + <p>a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/types.html#%28tech._c._type%29"><span class="techinside">C type</span></a>, which is a type description that tells +the FFI how to marshall between Racket and C.</p></li></ul> + +<p>C types are a crucial concept for the FFI. They range from relatively +simple types like <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> to more complicated +type constructors such as <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span>. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span></span> with two functions that +handle marshalling between C and Racket code.</p> + +<p>To make progress with our Cairo code, we need to create a drawing context from +the surface object <span class="RktWrap"><span class="RktSym">bt-surface</span></span> that we defined a while ago. The +relevant function in the Cairo docs is +<a href="https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-create"><span class="stt">cairo_create</span></a>, +which has the following type signature:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">NB:</span><span class="hspace">&nbsp;</span><span class="RktCmt">this</span><span class="hspace">&nbsp;</span><span class="RktCmt">is</span><span class="hspace">&nbsp;</span><span class="RktCmt">C</span><span class="hspace">&nbsp;</span><span class="RktCmt">code</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_create</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_surface_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*target</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a <span class="stt">cairo_surface_t</span> and +returns a pointer to a <span class="stt">cairo_t</span>. Let&rsquo;s start with a very simple C type +that matches up with this behavior: </div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span></p></div></div> + +<div class="SIntrapara">This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type.</div> + +<p>The following definition shows how to use this type to obtain a foreign +function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Then we can use <span class="RktWrap"><span class="RktSym">cairo-create</span></span> as an ordinary racket function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">ctx</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Interlude__more_type_safety)"></a>Interlude: more type safety</h1> + +<p>Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> types, it is easy +to accidentally misuse the function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">You may not want to actually run this</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a cairo_t is not a cairo_surface_t</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>To prevent such bad uses, it is good practice to use <span class="emph">tagged</span> pointer types +using <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>. Here are two example definitions that +correspond to the <span class="stt">cairo_t</span> and <span class="stt">cairo_surface_t</span> types from earlier:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">The leading underscores are mandatory</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>We can then redefine <span class="RktWrap"><span class="RktSym">cairo-create</span></span> with a better type, which will +prevent ill-typed calls:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">cairo_surface_t-&gt;C: argument is not non-null</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">`cairo_surface_t' pointer</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktErr">argument: #&lt;cpointer:cairo_t&gt;</span></p></td></tr></tbody></table></div> + +<p>Unfortunately our old definition of <span class="RktWrap"><span class="RktSym">ctx</span></span> doesn&rsquo;t have this tag:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#f</span></p></td></tr></tbody></table></div> + +<p>Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it&rsquo;s also possible to update +existing pointers with a tag like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-push-tag%21%29%29">cpointer-push-tag!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#t</span></p></td></tr></tbody></table></div> + +<p>Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).</p> + +<h1><a name="(part._.Macros_for_reducing_boilerplate)"></a>Macros for reducing boilerplate</h1> + +<p>Now let&rsquo;s start building the FFI bindings for the functions in the Cairo sample. +First, let&rsquo;s go ahead and look at all of the types for the sample functions +from the C API docs:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_cap_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">line_cap</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Starting with <span class="stt">cairo_move_to</span>, we can set up a definition like we did +with <span class="stt">cairo_create</span> before:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">"cairo_move_to"</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span> library that help reduce the +verbosity. Here&rsquo;s an alternative definition of <span class="RktWrap"><span class="RktSym">cairo-move-to</span></span> +using the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form from +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>As you can see, the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +<span class="RktWrap"><span class="RktSym">cairo_move_to</span></span>) you also don&rsquo;t need to supply the C name either.</p> + +<p>The definitions for <span class="stt">cairo_line_to</span>, <span class="stt">cairo_set_line_width</span>, and +<span class="stt">cairo_stroke</span> aren&rsquo;t very interesting, so I&rsquo;ll just include them +below without comment:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="stt">cairo_set_line_cap</span> case is more interesting because the type +<span class="stt">cairo_line_cap_t</span> is a C enumeration type. Racket&rsquo;s FFI comes with +convenience forms for defining enumeration types&#8212; + <wbr />though it&rsquo;s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here&rsquo;s a quote from the Barzilay and Orlovsky +paper on that:</p> + +<blockquote class="SubFlow"> + <p>Our design follows a simple principle: keep C-level +functionality to a minimum.</p></blockquote> + +<p>and specifically about enumerations:</p> + +<blockquote class="SubFlow"> + <p>For example, the C level part of our interface does not commit to a +specific implementation for enumerations &#8212; it simply exposes C integers.</p></blockquote> + +<p>To define an enumeration, we can use the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the <span class="stt">cairo_line_cap_t</span> type, it suffices to +just supply the cases as a list of symbols:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It&rsquo;s also possible to specify how the symbols map to integers more precisely +(see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> for those details).</p> + +<p>Given this type, we can specify the type for the line cap function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<h1><a name="(part._.Putting_it_all_together)"></a>Putting it all together</h1> + +<p>Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">30.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-cap</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">round</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object <span class="RktWrap"><span class="RktSym">bt</span></span>. To see the +results of what we drew, we can just evaluate <span class="RktWrap"><span class="RktSym">bt</span></span> at the REPL. But it&rsquo;s +a little nicer if we use the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> library to draw a frame around +it to distinguish it from the background:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>And we&rsquo;re done! Of course, there is a lot more to the FFI. For example, I haven&rsquo;t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I&rsquo;m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!</p> + + Gradual Typing Across the Spectrum + http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/?utm_source=Author-Asumu-Takikawa&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-18-gradual-typing-across-the-spectrum + Wed, 18 May 2016 07:58:56 UT + Asumu Takikawa + +<blockquote> + <p>Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. &mdash; Matthias Felleisen</p></blockquote> + +<p>Yesterday we hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found <a href="http://prl.ccs.neu.edu/gtp/pi2016/pi2016.html">here</a>). In case you aren&rsquo;t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.</p> + +<p>Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing&mdash;the ideas that cross language and platform barriers.</p> +<!-- more--> + +<p>There were a good number of both institutions and programming languages represented at the meeting, with researchers from all of <a href="http://cs.brown.edu/research/plt/">Brown University</a>, <a href="https://wonks.github.io/">Indiana University</a>, <a href="http://prl.ccs.neu.edu/">Northeastern University</a>, and the <a href="http://www.cs.umd.edu/projects/PL/">University of Maryland</a>. The languages that we work on cover a broad subset of the dynamically-typed languages: Clojure, JavaScript, R, Racket, Ruby, Pyret, and Python.</p> + +<div class="figure"><img src="/img/2016-day-slide-4.png" alt="" /> + <p class="caption"></p></div> + +<p>The specific research artifacts that were represented include <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>, <a href="https://github.com/plum-umd/rdl">RDL</a> (contracts for Ruby), <a href="http://plg.uwaterloo.ca/~dynjs/strongscript/">StrongScript</a>, <a href="http://typedclojure.org/">Typed Clojure</a>, and <a href="http://docs.racket-lang.org/ts-guide/index.html">Typed Racket</a>.</p> + +<p>In this blog post, I&rsquo;ll summarize some of the key research themes that were brought up at the meeting. Since I can&rsquo;t go into too much detail about every topic, I will link to the relevant research papers and other resources.</p> + +<p>At a high level, the talks covered four major facets of gradual typing: expressiveness, performance, usability, and implementation techniques.</p> + +<h2 id="expressiveness">Expressiveness</h2> + +<p>By expressiveness, I mean what kinds of language features a gradual type system supports and the richness of the reasoning that the type system provides. Since gradual typing is about augmenting existing dynamically-typed languages, a gradual type system should support the language features that programmers actually use.</p> + +<p>This is why recent implementations of gradual typing have focused on enabling object-oriented programming, since objects are widely used in nearly all dynamically-typed languages in use today. Unfortunately, since different languages have wildly different object systems, it&rsquo;s hard to compare research on gradual OO languages. Ben Chung is working to address this by coming up with a formal model that tries to unify various accounts of objects in order to better explain the design tradeoffs. His goal is to cover the core ideas in Reticulated Python, StrongScript, and Typed Racket.</p> + +<p>Of course, dynamically-typed languages have a lot more than OO features. Along these lines, I gave a talk on how at NU we&rsquo;re working to extend Typed Racket to cover everything from objects (my thesis topic), first-class modules (Dan Feltey&rsquo;s MS project), and higher-order contracts (Brian LaChance&rsquo;s MS project).</p> + +<p>On the other side, as programs get more complex, programmers may wish to write richer type specifications that provide even more guarantees. This makes gradual typing a wide spectrum that goes from completely untyped, fully typed, and then beyond to dependently typed. Andrew Kent and David Christiansen both presented work that takes gradual typing beyond ordinary typed reasoning with dependent types.</p> + +<p>Andrew presented an extension of Typed Racket that adds type refinements that can check rich properties (like vector bounds) that are found in real Racket code (see his RacketCon <a href="https://www.youtube.com/watch?v=ejFJIAsvdEg">talk</a> and recent <a href="http://arxiv.org/pdf/1511.07033.pdf">PLDI paper</a>). David Christiansen followed with a talk about adding dependent type theory to Typed Racket, which would allow correct-by-construction programming using a Nuprl-like proof system (he had a very cool GUI proof assistant demo in his slides!).</p> + +<h2 id="performance">Performance</h2> + +<div class="figure"><img src="/img/2016-day-slide-8.png" alt="" /> + <p class="caption"></p></div> + +<p>One of the key practical concerns about gradual typing is its performance overhead. It&rsquo;s a concern because in order to ensure type safety, a gradually-typed language implementation needs to install dynamic checks between the typed and untyped parts of a program. This catches any inconsistencies between the typed interfaces and how the untyped code may call into them.</p> + +<p>Ben Greenman gave an upbeat talk that set the stage for this topic, pointing out some key lessons that we&rsquo;ve learned about performance from building Typed Racket. The main idea he presented (also the topic of our <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016 paper</a>) is that to evaluate a gradual type system, you want to explore different ways of adding types to a program and see how much it costs. This evaluation effort started with Typed Racket, but he and Zeina Migeed are working on expanding it to Reticulated Python.</p> + +<p>From IU, Andre Kuhlenschmidt and Deyaaeldeen Almahallawi are exploring how ahead-of-time (AOT) compilation strategies could help reduce the cost of gradual typing. In particular, they are working on implementing <a href="https://github.com/deyaaeldeen/Schml">Schml</a>: a compiler from the gradually-typed lambda calculus to C.</p> + +<p>In addition to AOT compilation, the folks at IU are exploring tracing JIT compilation as a means to make gradual typing faster. More specifically, Spenser Bauman talked about Pycket, an alternative implementation of Racket that uses RPython/PyPy to dramatically lower the overhead of gradual typing (also see the <a href="https://www.youtube.com/watch?v=GOfIY8NHAqg">recording</a> of Spenser&rsquo;s talk on the topic at RacketCon and his <a href="http://homes.soic.indiana.edu/samth/pycket-draft.pdf">ICFP paper</a>).</p> + +<h2 id="usability">Usability</h2> + +<p>On the usability side, both Shriram Krishnamurthi and Ambrose Bonnaire-Sergeant made observations on what it takes to get gradual typing in the hands of real software developers.</p> + +<p>Shriram approached the topic from the angle of CS education, which is the focus of the <a href="http://www.pyret.org">Pyret</a> language, and shared what the Brown language group is working on. While Pyret doesn&rsquo;t exactly fit the mold of gradual typing, it&rsquo;s a close cousin since it&rsquo;s a dynamically-typed language that explicitly takes design cues from the best parts of statically-typed languages. That approach lets CS beginners think in terms of types (the approach spearheaded by <a href="http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html">HtDP</a> and <a href="http://www.bootstrapworld.org/">Bootstrap</a>) without having to battle a typechecker from the start.</p> + +<p>For professional software developers, a major concern with gradual typing is that writing type annotations may be a tedious and time intensive task. Ambrose, who is the creator of Typed Clojure, shared some preliminary work on how to cut down on the tedium by inferring gradual type annotations by instrumenting programs for a dynamic analysis. The hope is to be able to infer both recursive and polymorphic type annotations automatically from tests (you may also be interested in Ambrose&rsquo;s recent <a href="http://frenchy64.github.io/papers/esop16-short.pdf">ESOP paper</a> on Typed Clojure).</p> + +<h2 id="implementation-techniques">Implementation Techniques</h2> + +<p>Finally, several talks focused on alternative implementation techniques for gradual typing that provide a variety of software engineering benefits for implementers.</p> + +<p>From Maryland, Brianna Ren gave a talk on Hummingbird, a just-in-time typechecker for Ruby programs (also see the upcoming <a href="http://www.cs.umd.edu/~jfoster/papers/pldi16.pdf">PLDI paper</a> by Brianna and Jeff Foster). The basic idea is that it&rsquo;s hard to implement a traditional static typechecker for a language that heavily relies on metaprogramming, in which the fields/methods of classes may be rewritten at run-time. This is particularly tricky for frameworks like Ruby on Rails. Instead of checking types at compile-time, Hummingbird actually executes the typechecker at run-time in order to be able to accurately check programs that use run-time metaprogramming. To reduce overheads, she uses a cache for typechecking that is invalidated when classes are modified.</p> + +<p>Stephen Chang gave a very different view on metaprogramming in his talk, which focused on <em>implementing</em> typecheckers using metaprogramming (the <a href="http://docs.racket-lang.org/trivial/index.html">trivial</a> Typed Racket library is an offshoot of this work). His key idea is that typecheckers share many aspects with macro-based metaprogramming systems, such as the need to traverse syntax and annotate it with information. Since they share so much in common, why not just implement the typechecker as a macro? Stephen demonstrates that not only is this possible, but that it&rsquo;s possible to implement a wide variety of type system features this way including (local) type inference. The connection to gradual typing is that even a gradual type system can be implemented as a metaprogram by integrating the generation of dynamic checks into the macro transformation process.</p> + +<p>The last talk of the day (but certainly not the least), was by Michael Vitousek, who focused on the <em>transient</em> implementation of gradual typing (first described in his <a href="http://homes.soic.indiana.edu/mvitouse/papers/dls14.pdf">DLS paper</a>). Traditionally, gradual type systems have implemented their dynamic checks using <a href="https://en.wikipedia.org/wiki/Proxy_pattern">proxy</a> objects that wrap method implementations with both pre- and post-checks. Unfortunately, this implementation technique often conflicts with the underlying language. Since proxying changes the identity of an object it can interfere with object equality tests. Instead, the transient approach bakes the dynamic checks into and throughout the typed code to implement a &ldquo;defense in depth&rdquo; against inconsistencies with untyped code. The great thing about this implementation technique is that it doesn&rsquo;t demand any specialized support from the underlying language runtime and is therefore easy to port to other languages (like JavaScript).</p> + +<h2 id="conclusion">Conclusion</h2> + +<div class="figure"><img src="/img/2016-day-slide-3.png" alt="" /> + <p class="caption"></p></div> + +<p>Hopefully this blog post helps provide a better picture of the state of gradual typing research. The exciting thing about gradual typing is that it contains both interesting theoretical problems and also connects to the practical needs of software developers.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Ben-Chung.atom.xml b/blog/feeds/Author-Ben-Chung.atom.xml new file mode 100644 index 00000000..c1e84633 --- /dev/null +++ b/blog/feeds/Author-Ben-Chung.atom.xml @@ -0,0 +1,152 @@ + + + PRL Blog: Posts tagged 'Author: Ben Chung' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Ben-Chung-html + 2017-06-16T11:38:25Z + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + No Good Answers, Gradually Typed Object-Oriented Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-05-09-no-good-answers-gradually-typed-object-oriented-languages + 2017-05-09T14:04:31Z + 2017-05-09T14:04:31Z + + Ben Chung + <!-- more--> + +<p>Untyped code remains a real problem in practice, as a result of reduced performance and hindered readability. One approach to solve this problem is gradual typing.</p> + +<p>Gradual typing puts the onus on the developer to add type annotations, statically checks whatever type annotations have been written, and dynamically ensures that untyped code does not violate those annotations. A number of approaches have been put forward to try to achieve these objectives while retaining efficiency, semantic meaning, and the ability to actually type untyped code.</p> + +<p>I discussed three systems, all of which achieve the objective of typing untyped code in different ways, and all of which have different tradeoffs.</p> + +<p>Unofficial Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md</a></li></ul> + +<p>Code Examples:</p> + +<ul> + <li><a href="https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL">https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL</a></li></ul> + + Fall 2016 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + 2017-01-02T16:39:37Z + 2017-01-02T16:39:37Z + Ben Chung + Milo Davis + Ming-Ho Yee + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Ben-Chung.rss.xml b/blog/feeds/Author-Ben-Chung.rss.xml new file mode 100644 index 00000000..1541af9f --- /dev/null +++ b/blog/feeds/Author-Ben-Chung.rss.xml @@ -0,0 +1,136 @@ + + + + PRL Blog: Posts tagged 'Author: Ben Chung' + PRL Blog: Posts tagged 'Author: Ben Chung' + http://prl.ccs.neu.edu/blog/tags/Author-Ben-Chung.html + Fri, 16 Jun 2017 11:38:25 UT + Fri, 16 Jun 2017 11:38:25 UT + 1800 + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=Author-Ben-Chung&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + No Good Answers, Gradually Typed Object-Oriented Languages + http://prl.ccs.neu.edu/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/?utm_source=Author-Ben-Chung&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-09-no-good-answers-gradually-typed-object-oriented-languages + Tue, 09 May 2017 14:04:31 UT + Ben Chung + <!-- more--> + +<p>Untyped code remains a real problem in practice, as a result of reduced performance and hindered readability. One approach to solve this problem is gradual typing.</p> + +<p>Gradual typing puts the onus on the developer to add type annotations, statically checks whatever type annotations have been written, and dynamically ensures that untyped code does not violate those annotations. A number of approaches have been put forward to try to achieve these objectives while retaining efficiency, semantic meaning, and the ability to actually type untyped code.</p> + +<p>I discussed three systems, all of which achieve the objective of typing untyped code in different ways, and all of which have different tradeoffs.</p> + +<p>Unofficial Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md</a></li></ul> + +<p>Code Examples:</p> + +<ul> + <li><a href="https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL">https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL</a></li></ul> + + Fall 2016 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/?utm_source=Author-Ben-Chung&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + Mon, 02 Jan 2017 16:39:37 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Ben-Greenman.atom.xml b/blog/feeds/Author-Ben-Greenman.atom.xml new file mode 100644 index 00000000..784980b3 --- /dev/null +++ b/blog/feeds/Author-Ben-Greenman.atom.xml @@ -0,0 +1,4425 @@ + + + PRL Blog: Posts tagged 'Author: Ben Greenman' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Ben-Greenman-html + 2020-12-23T18:21:55Z + + Deep and Shallow Types + + urn:http-prl-ccs-neu-edu:-blog-2020-12-23-deep-and-shallow-types + 2020-12-23T18:21:55Z + 2020-12-23T18:21:55Z + + Ben Greenman + +<p>I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:</p> + +<ul> + <li><a href="http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020</a></li></ul> + +<p>To the PRL: thanks for a wonderful 6.5 years.</p> +<!-- more--> + +<h3 id="abstract">Abstract</h3> + +<blockquote> + <p>The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound&rsquo;&rsquo; mixed-typed languages from others that enforce type obligations locally rather than globally.</p> + <p>In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.</p> + <p>My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.</p> + <p>I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.</p></blockquote> + +<p>Next year, I&rsquo;ll be a <a href="https://cifellows2020.org">CI Fellow</a> at Brown.</p> + + Transient for Optional and Keyword Functions + + urn:http-prl-ccs-neu-edu:-blog-2020-11-12-transient-for-optional-and-keyword-functions + 2020-11-12T10:15:16Z + 2020-11-12T10:15:16Z + + Ben Greenman + +<p>A short adventure into the depths of optional and/or keyword functions in Racket.</p> +<!-- more--> + +<hr /> + +<p>Transient, or rather <em>the Transient semantics for a mixed-typed language</em>, is one way to let statically-typed code safely interact with untyped code. You can read all about it in <a href="http://hdl.handle.net/2022/23172">Michael Vitousek&rsquo;s 2019 dissertation</a> or <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my 2020 dissertation</a>, and you can see how it compares to other mixed-typed semantics <a href="http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">here</a>. The idea is to give up on <a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">behavioral type guarantees</a> and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called <em>shape checks</em>; for example:</p> + +<ul> + <li>if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;</li> + <li>if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and</li> + <li>if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.</li></ul> + +<p>Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type <strong>T</strong> into a shape check, and we have a function with type <strong>T</strong> that needs to check its inputs. The question is how to actually do the check in Racket v7.9.</p> + +<p>In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if <strong>T = (-&gt; Symbol Boolean)</strong> then we need to check the shape <strong>symbol?</strong> of the domain type <strong>Symbol</strong>:</p> + +<pre><code>;; source code +(: f (-&gt; Symbol Boolean)) +(define (f sym) + (eq? sym 'hola)) + +;; ===&gt; + +;; imaginary (but realistic) rewritten code +(define (f sym) + (assert sym symbol?) + (eq? sym 'hola))</code></pre> + +<p>A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type <strong>T</strong> accepts 1 to 3 inputs:</p> + +<pre><code>;; source code +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) +(define (g #:a a [b 'b] #:c [c #f]) + (if a b (if c 'left 'right))) + +;; ===&gt; + +;; imaginary, unrealistic rewritten code +(define (g #:a a [b 'b] #:c [c #f]) + (assert a boolean?) + (assert b symbol?) + (assert c void?) + (if a b (if c 'left 'right)))</code></pre> + +<p>Good &mdash; we basically know what we want. If the Racket core language had optional and keyword functions, then we&rsquo;d be done.</p> + +<p>But no, Racket expands these optional/keyword functions into primitive <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._lam))"><strong>lambda</strong></a> and <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._case-lam))"><strong>case-lambda</strong></a> forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.</p> + +<p>Let&rsquo;s keep digging.</p> + +<p>From now on, &ldquo;Shallow&rdquo; or &ldquo;Shallow TR&rdquo; refers to my implementation of Transient for Typed Racket (TR). We&rsquo;ll talk about Shallow instead of &ldquo;Transient&rdquo; in case future work reveals a better way to implement the Transient idea.</p> + +<h2 id="false-start-follow-the-type">False Start: Follow the Type</h2> + +<p>Beware &mdash; Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function <strong>g</strong> above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional <strong>#:c</strong> keyword lead to a shape-check failure because the variable <strong>c</strong> gets the default value <strong>#f</strong> instead of a void value. Concretely, the third assert from above fails:</p> + +<pre><code>(define (g #:a a [b 'b] #:c [c #f]) + .... + (assert c void?) ;; fails if c is the #f default value + ....)</code></pre> + +<p>The problem arises from subtyping. According to the annotations, the function <strong>g</strong> has an external type that is less precise than the internal type that validates the function body:</p> + +<pre><code>;; external type T +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) + +;; internal type T2, subtype of external (T2 &lt;: T), validates body +(: g (-&gt;* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))</code></pre> + +<p>Thanks to this external / internal distinction, the following easy rewrite idea, <em>Solution 0</em>, fails. Despite the failure, this first solution is a useful starting point for a success.</p> + +<h4 id="solution-0-step-1-mimic-the-typechecker">Solution 0, Step 1: Mimic the Typechecker</h4> + +<p>Shallow TR uses the same type checker as classic <em>Deep</em> TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.</p> + +<p>Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.</p> + +<p>Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (<a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-expr-unit.rkt#L274-L295">typecheck/tc-expr-unit.rkt</a>). The omitted code (<strong>&hellip;.</strong>) does actual type checking:</p> + +<pre><code>(define (tc-expr/check/internal form expected-type) + .... + (syntax-parse form + #:literal-sets (kernel-literals tc-expr-literals) + .... + [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^) + ....] + [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>Ok! Those two patterns say a lot about the expansion of optional and keyword functions:</p> + +<ol> + <li>Both forms expand to a <strong>let-values</strong> that binds one function <strong>fun</strong>.</li> + <li>TR uses the syntax classes <strong>kw-lambda^</strong> and <strong>opt-lambda^</strong> to tell these particular <strong>let-values</strong> apart from others.</li></ol> + +<p>Shallow TR can use exactly these patterns to recognize optional/keyword functions.</p> + +<h4 id="solution-0-step-2-parse-the-domain-type">Solution 0, Step 2: Parse the Domain Type</h4> + +<p>Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function&rsquo;s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:</p> + +<pre><code> [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/type-table.rkt#L80"><strong>type-of</strong></a> function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, <strong>fun</strong>, has a type. Great.</p> + +<p>Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:</p> + +<ul> + <li>The type of <strong>fun</strong> could be a straightforward <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L693"><strong>Fun type</strong></a>, but it could also be a: <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L701"><strong>DepFun type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L521"><strong>Poly type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L531"><strong>PolyDots type</strong></a>, or even a <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L900"><strong>Union type</strong></a>.</li> + <li>Each part of the domain type corresponds to one parameter of the <strong>fun</strong> expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in <strong>fun</strong>, or the mandatory keywords?</li></ul> + +<h4 id="solution-0-step-3-insert-a-shape-check">Solution 0, Step 3: Insert a Shape Check</h4> + +<p>Once we have the target <strong>fun</strong> expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse <strong>fun</strong> to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting <strong>fun</strong> goes something like this:</p> + +<pre><code>(syntax-parse fun + [(#%plain-lambda formals . body) + #:with (shape-check ...) + (make-shape-checks #'formals (type-of fun)) + #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])</code></pre> + +<p>The rewritten function executes shape checks immediately, and then proceeds with the <strong>body</strong> after validating each actual parameter.</p> + +<h2 id="on-the-trail-optkey-expansion">On the Trail: optkey Expansion</h2> + +<p>Our <em>Solution 0</em> fails because the type of the <strong>fun</strong> expression that it gets from the type-checked code is an external type. In terms of the <strong>g</strong> function from above, <em>Solution 0</em> uses the type <strong>Void</strong> instead of the internal type <strong>(U Void #f)</strong> to check the <strong>c</strong> parameter. To get internal types, we need to look closer at <strong>fun</strong> and the rest of the optional/keyword expansion.</p> + +<p>Let&rsquo;s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.</p> + +<p>If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function <strong>X</strong> create a module <strong>test.rkt</strong> like this:</p> + +<pre><code>#lang racket/base + +(define _ignore + (let () + X + (void)))</code></pre> + +<p>Invoke the expander with <code>raco expand test.rkt &gt; test.rkt.txt</code> and explore the generated <strong>.txt</strong> file.</p> + +<h3 id="example-1-mandatory-keyword">Example 1: mandatory keyword</h3> + +<p>The source is a function with one mandatory positional argument and one optional positional argument.</p> + +<pre><code>(lambda (x [y 0]) + (+ x y))</code></pre> + +<p>Expansion generates a <strong>case-lambda</strong> that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function <strong>F</strong> that expects two arguments, resolves defaults in a different way, and executes the function body.</p> + +<pre><code>(let-values (((F) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F x '0)) + ((x y1) (#%app F x y1))))</code></pre> + +<p>Note: the expression <strong>(if &rsquo;#f &rsquo;0 y1)</strong> in the generated <strong>F</strong> function is equal to <strong>y1</strong> alone. In general, the <strong>if</strong> is for default expressions. (<a href="https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/">Unlike Python</a>, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a <strong>#f</strong> test. A general-purpose optimizer can remove this test before the code runs.</p> + +<h3 id="example-2">Example 2:</h3> + +<p>The source is a function with one mandatory positional argument and one mandatory keyword argument:</p> + +<pre><code>(lambda (x #:y y) + (+ x y))</code></pre> + +<p>Expansion generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> expects a list of keywords, a list of arguments, and a final argument. The purpose of <strong>F1</strong> is to organize a call to <strong>F0</strong>.</li> + <li><strong>lifted/2</strong> is the constructor for a generated struct type. Other functions help the struct call <strong>F1</strong>. Nevermind the details; I don&rsquo;t fully understand them either.</li></ul> + +<p>The important piece for Shallow TR is the <strong>F0</strong> function because the goal of rewriting is to protect the original function body against untyped inputs.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) y1)) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y1) (#%app car given-args))) + (#%app F0 y1 x3))))) + (#%app + lifted/2 + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l2571) given-kws)) + (if (#%app pair? l2571) + (if (#%app eq? (#%app car l2571) '#:y) + (#%app null? (#%app cdr l2571)) + '#f) + '#f)) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + '(#:y) + '(#:y))))</code></pre> + +<h3 id="example-3">Example 3:</h3> + +<p>The source is a function with one mandatory positional argument and one optional keyword argument:</p> + +<pre><code>(lambda (x #:y [y 0]) + (+ x y))</code></pre> + +<p>Expansion again generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments, resolves the optional default, and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> calls <strong>F0</strong></li> + <li>At the bottom, there are two <strong>case-lambda</strong> functions that call <strong>F1</strong></li></ul> + +<p>Again, the <strong>F0</strong> function is the focal point for Shallow TR rewriting.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y2) (#%app pair? given-kws))) + (let-values (((y1) + (if y2 (#%app car given-args) '0))) + (#%app F0 y1 x3)))))) + (#%app + make-optional-keyword-procedure + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l1571) given-kws)) + (let-values (((l1571) + (if (#%app null? l1571) + l1571 + (if (#%app eq? (#%app car l1571) '#:y) + (#%app cdr l1571) + l1571)))) + (#%app null? l1571))) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + null + '(#:y) + (case-lambda + ((x) (#%app F1 null null x))))))</code></pre> + +<h2 id="solution-the-shallow-tr-rewrite-strategy">Solution: The Shallow TR Rewrite Strategy</h2> + +<p>All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a <strong>let-values</strong> form:</p> + +<pre><code>(let-values (((f) fun)) . body)</code></pre> + +<p>Furthermore, the generated <strong>fun</strong> is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the <strong>fun</strong> from <em>Example 3</em> again; it has formal parameters for the keyword arg. and the mandatory arg., and one <strong>let-values</strong> to resolve each parameter:</p> + +<pre><code> (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y)))))</code></pre> + +<p>Another experiment with <strong>type-of</strong> shows that the right-hand side of each <strong>let-values</strong> has an internal type annotation. Excellent! Both <strong>(type-of x3)</strong> and <strong>(type-of (if &rsquo;#f &rsquo;0 y1))</strong> are the right types for shape checks. Shallow TR can:</p> + +<ul> + <li>inspect the <strong>let-values</strong> one-by-one;</li> + <li>convert the type of each right-hand expression to a shape predicate; and</li> + <li>rewrite each right-hand <strong>expr</strong> into <strong>(assert expr shape?)</strong>.</li></ul> + +<p>This should work! In fact, we can do slightly better:</p> + +<ul> + <li>when the right-hand expression is a conditional <strong>(if test default-expr supplied-arg)</strong></li> + <li>then Shallow only needs to check the supplied arg: <strong>(if test default-expr (assert supplied-arg shape?))</strong></li></ul> + +<p>Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.</p> + +<h2 id="a-problem-with-methods-and-a-bugfix">A Problem with Methods and a Bugfix</h2> + +<p>Currently, Shallow TR rewrites optional and keyword functions using the <strong>let-values</strong> plan described above. Each formal parameter has one <strong>let-values</strong> binding, and the type on each bound expression defines the shape check.</p> + +<p>Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we <a href="https://github.com/racket/racket/pull/3182">fixed the class expander</a>.</p> + +<p>First, here is a class with one method that runs correctly. The method <strong>f</strong> accepts an optional positional argument <strong>x</strong>; the default value of <strong>x</strong> is the current value of the field <strong>my-num</strong> (fields are mutable):</p> + +<pre><code>(define c0% + (class object% + (super-new) + (field (my-num 2)) + (define/public (f [x my-num]) + (+ x x))))</code></pre> + +<p>Second, here is a similar method that fails. This time, the default is an immediate value <strong>2</strong>:</p> + +<pre><code>(define c1% + (class object% + (super-new) + (define/public (f [x 2]) + (+ x x))))</code></pre> + +<p>Running a call <strong>(send o1 f)</strong> used to raise a shape-check failure about a strange value:</p> + +<blockquote> + <p>shape error: Expected a real number, got <code>#&lt;unsafe-undefined&gt;</code></p></blockquote> + +<p>What is going on?</p> + +<p>It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:</p> + +<pre><code>(lambda (x [y z]) + (+ x y))</code></pre> + +<p>Expansion generates a function <strong>F0</strong> that checks for the undefined value, and an outer <strong>case-lambda</strong> that supplies undefined when the default is needed:</p> + +<pre><code>(let-values (((F0) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) + (if (#%app eq? y1 unsafe-undefined) + z + y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F0 x unsafe-undefined)) + ((x y1) (#%app F0 x y1))))</code></pre> + +<p>That&rsquo;s the normal way that <strong>unsafe-undefined</strong> shows up: the <a href="https://github.com/racket/racket/blob/c0ff11e27bd28e070c20b7a9b0f7365f8f2b665a/racket/collects/racket/private/kw.rkt">expander for optional/keyword functions</a> looks for default expressions vs. default values and uses the undefined value for expressions.</p> + +<p>Three other facts conspired to make the problem with optional methods:</p> + +<ol> + <li>Typed Racket also looks for default expressions vs. default values (search for <strong>immediate-default</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/base-env/annotate-classes.rkt">here</a>). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the <strong>unsafe-undefined</strong> value (search for <strong>-Unsafe-Undefined</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/kw-types.rkt">here (kw)</a> and <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-lambda-unit.rkt">here (opt)</a>).</li> + <li>The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.</li> + <li>Shallow TR pushes default expression checks <strong>(if test default-expr supplied-arg)</strong> to the <strong>supplied-arg</strong> instead of wrapping the whole <strong>if</strong> form.</li></ol> + +<p>In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound&mdash;but harmless because the false assumption was guarded by an <strong>if</strong> test for <strong>unsafe-undefined</strong>. Running Shallow TR revealed the unsoundness with its eager shape check.</p> + +<p>Again, the resolution was to fix the class expander (<a href="https://github.com/racket/racket/pull/3182">racket/racket #3182</a>). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.</p> + +<h2 id="lessons">Lessons</h2> + +<ol> + <li>Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.</li> + <li>Digging into the expansion is sometimes necessary. There are at least three places that do so&mdash;the class expander, TR, and Shallow TR&mdash;and unfortunately they all need to cooperate.</li> + <li>The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my dissertation</a> lists them all.</li></ol> + + Transient Answers Old Questions + + urn:http-prl-ccs-neu-edu:-blog-2020-10-15-transient-answers-old-questions + 2020-10-15T13:32:12Z + 2020-10-15T13:32:12Z + + Ben Greenman + +<p>Several old questions from the Typed Racket mailing list have new and simple answers under a &ldquo;transient&rdquo; Typed Racket.</p> +<!-- more--> + +<hr /> + +<p>For the past few months, I&rsquo;ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the <a href="https://github.com/racket/typed-racket/pull/952">RFC</a> and <a href="https://github.com/racket/typed-racket/pull/948">pull request</a>.</p> + +<p>The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that&rsquo;s about it. By contrast, types are much stronger in classic Typed Racket.</p> + +<p>Shallow Racket&rsquo;s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the <a href="https://groups.google.com/g/racket-users">Racket mailing list</a>. There were a surprising number of questions that went like this:</p> + +<blockquote> + <p><strong>Q.</strong> Hey, I ran a program expecting <em>X</em> to happen, but <em>Y</em> happened instead. Is this a bug?</p> + <p><strong>A.</strong> No, Typed Racket has to do <em>Y</em> because of its strong types.</p></blockquote> + +<p>&hellip; but changing to shallow types gives the <em>X</em> behavior! Here are their stories.</p> + +<p>Going forward, <strong>Deep</strong> refers to normal Typed Racket and <strong>Shallow</strong> refers to Shallow Typed Racket.</p> + +<hr /> + +<h2 id="higher-order-value-as-any">Higher-Order Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ">groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ</a></p> + +<h4 id="on-20180416-mailoo-wrote">On 2018&ndash;04&ndash;16, <em>mailoo</em> wrote:</h4> + +<blockquote> + <p> I play a little with the &ldquo;Any&rdquo; type (due to &lsquo;dynamic-require&rsquo; which return Any), and I&rsquo;m not able to cast them back in a function.</p> + <p> I (over) simplify my question with this little program :</p></blockquote> + +<pre><code>(: p Any) +(define (p i) (displayln i)) + +; Here I want to get back my function +(define proc (cast p (-&gt; Integer Void))) +(proc 2) </code></pre> + +<blockquote> + <p> but I get this error when I try to execute the function :</p></blockquote> + +<pre><code>; contract violation +; Attempted to use a higher-order value passed as `Any` in untyped code: #&lt;procedure:p&gt; </code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> raises an error because it must enforce the <code>Any</code> type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> prints 2 and returns void. No error. Same goes for dynamic-require.</p> + +<hr /> + +<h2 id="parametric-contract-affects-untyped-code">Parametric Contract Affects Untyped Code</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ">groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ</a></p> + +<h4 id="on-20191215-john-clements-wrote">On 2019&ndash;12&ndash;15, John Clements wrote:</h4> + +<blockquote> + <p> It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:</p></blockquote> + +<pre><code> #lang typed/racket + + (require/typed racket/list + [index-of (All (T) ((Listof T) T -&gt; (U False Natural)))]) + + (index-of '(n s e w) 'n) ;; returns... #f? </code></pre> + +<blockquote> + <p> In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).</p> + <p> I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces parametricity for <code>All</code> types, and this throws off the equality function that index-of uses internally.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> returns 0.</p> + +<p>ps John, thanks very much for working on <a href="https://adventofcode.com">Advent of Code</a> and mailing the list!</p> + +<hr /> + +<h2 id="unable-to-protect-opaque-value-as-any">Unable to Protect Opaque Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ">groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ</a></p> + +<h4 id="on-20191211-marc-kaufmann-wrote">On 2019&ndash;12&ndash;11, Marc Kaufmann wrote:</h4> + +<blockquote> + <p>I have one file called <code>type-test.rkt</code> with the following</p></blockquote> + +<pre><code>#lang typed/racket + +(require (only-in typed/web-server/http response/xexpr response)) + +(provide f2) + +(: f2 (-&gt; (U response Any))) +(define (f2) + (define x '(body (h1 "Try it"))) + (: resp response) + (define resp (response/xexpr x)) + resp)</code></pre> + +<blockquote> + <p>Then I have another <em>untyped</em> file for a servlet:</p></blockquote> + +<pre><code>#lang racket + +(require "type-test.rkt" + web-server/servlet + web-server/servlet-env) + +(define (start req) + (f2)) + +(serve/servlet start + #:servlet-regexp #rx"" + #:launch-browser? #false + #:port 8080)</code></pre> + +<blockquote> + <p>Notice that I am telling [f2] that <code>resp</code> is of type <code>response</code>. Yet, when I run the server with <code>start</code> [&hellip;.] I get the following result:</p> + <p>(f2): Error, see below.</p> + <p>The error is:</p></blockquote> + +<pre><code>f2: broke its own contract + any-wrap/c: Unable to protect opaque value passed as `Any` + value: #&lt;response&gt; + in: the range of + (-&gt; Any)</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the <code>Any</code> type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.</p> + +<hr /> + +<h2 id="type-inference-installs-a-precise-type">Type Inference Installs a Precise Type</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ">groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ</a></p> + +<h4 id="on-20200214-john-clements-wrote">On 2020&ndash;02&ndash;14, John Clements wrote:</h4> + +<blockquote> + <p>I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.</p> + <p>Here’s a program:</p></blockquote> + +<pre><code>#lang typed/racket + +(define-type Store (Mutable-HashTable Integer Value)) +(define-type Value (U Real Boolean String)) + +(define top-store + (cast + (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f))) + Store)) + +(hash-set! top-store 5 1234)</code></pre> + +<blockquote> + <p>It fails with this error:</p></blockquote> + +<pre><code>contract violation +expected: (or/c (and/c byte? positive?) #t #f) +given: 1234 +in: the values of +the 3rd conjunct of +(and/c hash? + hash-mutable? + (hash/c exact-integer? + (or/c (and/c byte? positive?) #t #f) + #:immutable #f))</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p>First off, <strong>Deep</strong> runs fine after swapping <code>cast</code> for <code>ann</code>.</p> + +<p>Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.</p> + +<p>The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.</p> + +<hr /> + +<h2 id="same-arity-functions-in-a-case-lambda">Same-Arity Functions in a Case Lambda</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ">groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ</a></p> + +<h4 id="on-20190705-ryan-kramer-wrote">On 2019&ndash;07&ndash;05, Ryan Kramer wrote:</h4> + +<blockquote> + <p>In the code below, can <code>maybe-car</code> have the given type [&hellip;.]?</p></blockquote> + +<pre><code>#lang typed/racket + +(module untyped racket + (provide maybe-car) + (define (maybe-car x) + (cond + [(pair? x) (car x)] + [else x]))) + +(require/typed + 'untyped + [maybe-car (All (a b) (case-&gt; + (-&gt; (Pairof a b) a) + (-&gt; a a)))])</code></pre> + +<blockquote> + <p>[Current error:]</p></blockquote> + +<pre><code>Type Checker: + Type (All (a b) (case-&gt; (-&gt; (Pairof a b) a) (-&gt; a a))) + could not be converted to a contract: + function type has two cases of arity 1</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the type with a Racket <code>or/c</code> contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don&rsquo;t know which to apply to an incoming function.</p> + +<p>Note, the &ldquo;Type Checker&rdquo; error message is much better than what or/c would give on its own.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> simply checks that maybe-car accepts both arities inside the case-&gt; type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.</p> + +<hr /> + +<h3 id="immutable-type-affects-untyped-code">Immutable Type Affects Untyped Code</h3> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ">groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ</a></p> + +<h4 id="on-20200217-bertrand-augereau-wrote">On 2020&ndash;02&ndash;17, Bertrand Augereau wrote:</h4> + +<blockquote> + <p>Hello everybody, I&rsquo;m trying to gradually type my script to make it a proper app (yes I&rsquo;m a static-ish guy) and I have an issue (Racket 7.6 CS).</p></blockquote> + +<pre><code>; racket_mod.rkt: +#lang racket + +(provide (struct-out s)) +(provide list-of-s) +(provide set-list-of-s!) + +(struct s (a)) +(define list-of-s '()) +(define (set-list-of-s! los) + (set! list-of-s los))</code></pre> + +<pre><code>; racket_mod_typed.rkt: +#lang typed/racket + +(provide (struct-out s2)) +(provide list-of-s2) +(provide set-list-of-s2!) + +(struct s2 ([a : Natural])) +(define list-of-s2 '()) +(define (set-list-of-s2! [los : (Listof s2)]) + (set! list-of-s2 los))</code></pre> + +<pre><code>; racket_main.rkt: +#lang racket + +(require "racket_mod.rkt") +(require "racket_mod_typed.rkt") + +(define los (list (s 1) (s 2))) +(set-list-of-s! los) +(displayln list-of-s) + +(define los2 (list (s2 1) (s2 2))) +(set-list-of-s2! los2) +(displayln list-of-s2)</code></pre> + +<blockquote> + <p>list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces the type of <code>list-of-s2</code> with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.</p> + +<p>Here&rsquo;s a step-by-step breakdown:</p> + +<ol> + <li>the typed module creates an empty list-of-s2</li> + <li>the main module imports the list and receives a new copy</li> + <li>the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable</li> + <li>the main module reads from its copy &mdash; and it&rsquo;s still empty</li></ol> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> lets the original list travel to untyped code. There are no contracts in the way.</p> + +<h2 id="discussion">Discussion</h2> + +<p>Wow! It&rsquo;s great to see that Shallow Racket works &ldquo;as expected&rdquo; on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.</p> + +<p>If you have a similar experience with a deep-types error, let me know.</p> + +<p>Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error &mdash; and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.</p> + +<p>Shallow Typed Racket is coming soon. Follow the <a href="https://github.com/racket/typed-racket/pull/948">pull request</a> or watch the Racket release notes for news.</p> + +<h3 id="links">Links</h3> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">Larger example</a> where Shallow misses an error that Deep catches</li> + <li>Michael M. Vitousek <a href="http://hdl.handle.net/2022/23172">invented</a> the Transient semantics and implemented it in <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>.</li> + <li>My <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-thesis-2020">upcoming dissertation</a> has lots more to say about Shallow Typed Racket.</li></ul> + +<p><em>Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.</em></p> + + The Typed Racket Optimizer vs. Transient + + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + 2020-01-15T12:16:35Z + 2020-01-15T12:16:35Z + + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> + + PRL Offsite 2019 Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2019-12-12-prl-offsite-2019-retrospective + 2019-12-12T12:51:53Z + 2019-12-12T12:51:53Z + Ben Greenman + Olek Gierczak + + Ben Greenman, Olek Gierczak + +<p>On November 11th 2019, the PRL had a private offsite meeting at the <a href="https://mtwyouth.org">More Than Words bookstore</a> in downtown Boston. For future offsite organizers, this post records what happened and how.</p> +<!-- more--> + +<h2 id="early-planning-goals">Early Planning, Goals</h2> + +<p>Every Fall, the PRL holds a kickoff meeting to assign <a href="http://prl.ccs.neu.edu/contact">roles</a> for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.</p> + +<blockquote> + <p>Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the <a href="http://www.warrencenter.com">Warren Conference Center</a> is <a href="https://www.northeastern.edu/events/northeastern-owned-off-campus-venues">Northeastern University&rsquo;s only franchised</a> off-campus venue.</p></blockquote> + +<p>The <strong>motivation</strong> for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students &mdash; both to get to know the group, and to become known. They&rsquo;re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">Spring 2017 edition</a> of HOPL.</p> + +<p>Our primary <strong>goal</strong> this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.</p> + +<p>In light of the main goal, we prefer to call the event an &ldquo;offsite&rdquo; (or &ldquo;offsite meeting&rdquo;) rather than a &ldquo;retreat&rdquo; because the target was an informative day rather than a relaxing one.</p> + +<h2 id="booking-and-logistics">Booking and Logistics</h2> + +<p>Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that&rsquo;s all we had.</p> + +<h3 id="date--nov-11">Date = Nov 11</h3> + +<p>In the beginning, we chose Veterans&rsquo; Day (2019&ndash;11&ndash;11) and Northeastern <a href="https://registrar.northeastern.edu/app/uploads/2019-2020-University-Wide-Calendar-List.pdf">reading day</a> (2019&ndash;12&ndash;05) as possible dates. We ended up with Veterans&rsquo; Day.</p> + +<p>A small number of lab members were opposed to Veterans&rsquo; Day. They gave two reasons: the Fall semester is especially busy, and Veterans&rsquo; Day is a federal holiday.</p> + +<h3 id="venue--mtw">Venue = MTW</h3> + +<p>The first venue we tried was the <a href="http://www.warrencenter.com">Warren Conference Center</a> in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the <a href="https://www.khoury.northeastern.edu/people/chelsea-smith/">Khoury Events Team</a> to reach out on our behalf. They returned with an email address and <a href="/blog/static/warren-center-meetings-and-retreats.pdf">event menu</a> (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.</p> +<!-- Donald Pepple (Northeastern) made contact with Christine Barisano (Framingham)--> + +<p>Second, we contacted the <a href="https://alumni.northeastern.edu/about/alumni-center/">Northeastern University Alumni Center</a>. They were not open on Veterans&rsquo; Day 2019.</p> + +<p>Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: <a href="https://mtwyouth.org">More Than Words</a>.</p> + +<p>More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!</p> + +<blockquote> + <p>More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you&rsquo;ll see a mix of &ldquo;employees&rdquo; and &ldquo;volunteers&rdquo; helping out.</p></blockquote> + +<p>Booking was complicated, though, by the fact that Northeastern requires <a href="http://catalog.northeastern.edu/graduate/health-sciences/academic-policies-procedures/liability-insurance/">liability insurance</a> for all off-campus events. If <strong>you</strong> are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.</p> + +<p>Here are more facts about MTW:</p> + +<ul> + <li>the theater space seats 30 with lots of extra space</li> + <li>the two long tables in the reading room can seat about 20</li> + <li>the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor</li> + <li>there&rsquo;s no whiteboard, but MTW can supply an easel stand</li> + <li>much of the furniture in store is antique and for sale, so be careful using it &mdash; both to avoid damaging it, and because antiques can be oddly-shaped</li> + <li>the bookstore was closed on Veterans&rsquo; Day, but we were able to have our event and buy books</li> + <li>MTW may have fridge space to temporarily hold leftovers</li> + <li>closest T station: <a href="https://www.mbta.com/stops/place-tumnl">Tufts Medical Center</a></li></ul> + +<h3 id="keynote--none">Keynote = None</h3> + +<p>The original, ambitious plan was to invite two keynote speakers &mdash; one working in industry and one from academia &mdash; to enrich the offsite with new knowledge. And because this was the PRL&rsquo;s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)</p> + +<p>We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.</p> + +<p>Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).</p> +<!-- to future planners: we invited Kathi Fisler; she had grant-writing plans for that day and would be interested in coming to a future offsite--> + +<h3 id="food-coffee-tea">Food, Coffee, Tea</h3> + +<p><a href="https://flourbakery.com/">Flour Bakery + Cafe</a> provided lunch. For most days, you can order from Flour using an <a href="https://flourbakery.com/menu/catering/catering-information/">online form</a>. For holidays, you may need to send an email &mdash; that&rsquo;s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan &amp; gluten-free <em>everything spiced</em> salads.</p> + +<p><a href="https://www.trycuppacoffee.com/location/">Cuppacoffee</a> provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything &mdash; the order + napkins, milk, cream cheese, butter, and knives &mdash; ourselves because Cuppacoffee is very close to MTW.</p> + +<p><a href="http://www.cmartboston.com/">CMart</a> sold us water and juices. (There is also a Whole Foods near MTW.)</p> + +<p>At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour&rsquo;s <a href="https://youtu.be/kIbhckqanHI">sticky buns</a>.</p> + +<h3 id="event-program">Event Program</h3> + +<p>The following assumptions/goals constrained the schedule for the day:</p> + +<ul> + <li>give all 17 on-campus Ph.D. students a talk slot; talks must <strong>either</strong> communicate one technical idea from recent work <strong>or</strong> say what led the speaker to apply to a PL Ph.D. program</li> + <li>allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)</li> + <li>do not make the audience sit for more than 1 hour at a time</li> + <li>maximize the number and length of breaks (to encourage discussions)</li> + <li>include a relevant and fun welcome activity</li> + <li>save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions</li></ul> + +<p>We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.</p> + +<p>The activity was <em>codewalks</em>. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition/blob/master/src/main/java/com/seriouscompany/business/java/fizzbuzz/packagenamingpackage/impl/math/arithmetics/IntegerDivider.java">IntegerDivider</a> class from a silly FizzBuzz implementation and a solution to the <a href="https://en.wikipedia.org/wiki/Dutch_national_flag_problem">Dutch national flag problem</a>). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and <a href="https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/">egoless programming</a>.</p> + +<p>The talks ran conference-style. One organizer played &ldquo;session chair&rdquo; to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.</p> + +<p>For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.</p> + +<h2 id="looking-back-at-the-details">Looking Back at the Details</h2> + +<p>Reading Day in the Spring may be a good, free date for future retreats.</p> + +<p>We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.</p> + +<p>The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we&rsquo;d keep the morning breaks short &mdash; just enough to use the restroom and grab a coffee &mdash; and aim for 30-min breaks in the afternoon.</p> + +<p>Groups of three 15-minute talks worked well, but groups of four talks might be equally good.</p> + +<p>Perhaps each speaker should get the chance to pick a talk length. <a href="https://nepls.org/">NEPLS</a>, for example, allows a choice between 5-min. and 30-min. talks.</p> + +<p>The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick &ldquo;topics&rdquo; beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.</p> +<!-- to future organizers: the panel began asking faculty for a mistake in their career and wound up with a defensive flavor--> + +<h2 id="looking-back-overall">Looking Back, Overall</h2> + +<p>This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.</p> + +<p>There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time &mdash; keep the same date &amp; program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.</p> + +<p>With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.</p> + + Complete Monitors for Gradual Types + + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + 2019-10-31T21:58:26Z + 2019-10-31T21:58:26Z + + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + Forgetful and Heedful contracts + + urn:http-prl-ccs-neu-edu:-blog-2019-04-07-forgetful-and-heedful-contracts + 2019-04-07T23:15:11Z + 2019-04-07T23:15:11Z + + Ben Greenman + +<p><em>Forgetful</em> and <em>heedful</em> are two methods for space-efficient contracts developed by <a href="http://www.cs.pomona.edu/~michael/">Michael Greenberg</a> in <a href="https://arxiv.org/abs/1410.2813">2014</a>. These methods were born in the shadow of a third method, <em>eidetic</em>, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.</p> +<!-- more--> + +<hr /> + +<p>Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of <a href="https://www2.ccs.neu.edu/racket/pubs/popl11-dfff.pdf">three parties</a>: the API code, the client code, and the contract between them.</p> + +<p>For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:</p> + +<pre><code> #lang racket/base + + (module sort racket + (provide + (contract-out + [quicksort + (-&gt; (vectorof point/c) void?)])) + + (define point/c (vectorof integer?)) + + (define (quicksort points) + ....)) + + (module client racket + (require (submod ".." sort)) + (quicksort '())) + + (require 'client)</code></pre> + +<pre><code>quicksort: contract violation; + expected a vector + given: '() + in: the 1st argument of + (-&gt; (vectorof (vectorof integer?)) void?) + contract from: + (file.rkt sort) + blaming: (file.rkt client) + (assuming the contract is correct)</code></pre> + +<p>That covers the basics. For an extended introduction to contracts, visit <a href="https://docs.racket-lang.org/guide/contracts.html">The Racket Guide</a>.</p> + +<p>The quicksort example and the related figures are from the paper <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts: Fixing a Pathology of Gradual Typing</em></a></p> + +<h3 id="classic-contracts-and-space-efficiency">Classic contracts and &ldquo;Space Efficiency&rdquo;</h3> + +<p>The <code>(vectorof point/c)</code> contract used above describes a possibly-mutable array whose elements match the <code>point/c</code> contract. Since the array can be mutated, this contract has implications for two parties:</p> + +<ol> + <li>the client module must supply a good array, and</li> + <li>the sorting module must not insert a bad element.</li></ol> + +<p>To enforce the second condition, the <code>vectorof</code> contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:</p> + +<pre><code>(quicksort (vector (vector 4 4) + (vector 2 2) + (vector 1 1) + (vector 3 3)))</code></pre> + +<p>After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a <strong>solid</strong> blue bar for the <strong>write</strong> checks against the sort module and a <em>striped</em> blue bar for the <em>read</em> checks against the client.</p> + +<p><img src="/img/vector-chaperone-0.png" alt="A wrapped vector" /></p> + +<p>In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted &mdash; because every <strong>write</strong> and <em>read</em> applies a contract to the element.</p> + +<p><img src="/img/vector-chaperone-1.png" alt="Layers of element wrappers" /></p> + +<p>On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.</p> + +<p>Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and &ldquo;indirecting&rdquo; future writes/reads through wrappers. These space and time costs can add up.</p> + +<blockquote> + <p>&ldquo;on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times&rdquo; &mdash; <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts</em></a></p></blockquote> + +<p>To fix the problem, researchers have been exploring <em>space-efficient</em> implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and <em>eidetic</em>, <em>forgetful</em>, and <em>heedful</em> are his names for three implementations.</p> + +<p>(Although the goal of this post is to promote <em>forgetful</em> and <em>heedful</em>, we will review all three.)</p> + +<h3 id="eidetic-space-efficiency">Eidetic space-efficiency</h3> + +<p>The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a <em>merge</em> operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal &ldquo;wrapping&rdquo; contracts and their size is bounded by the number (and height) of source-code contracts in the program.</p> + +<p>An eidetic contract is an <code>N</code>-ary tree (for <code>N &gt; 0</code>):</p> + +<ul> + <li>each node represents a higher-order contract combinator, such as <code>vectorof</code></li> + <li>the <code>N</code> children of a node represent the different interactions that the value supports</li> + <li>each leaf is a list of non-higher-order, or <em>flat</em>, contracts</li></ul> + +<p>For example, the <code>(vectorof point/c)</code> source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf">Collapsible Contracts</a> paper has an illustration. Each tree node represents a <code>vectorof</code> contract; these nodes have <code>N=2</code> children because vectors support reads and writes.</p> + +<p>A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists &mdash; if an implementation comes with a <code>contract-stronger?</code> predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.</p> + +<p>Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should <code>contract-stronger?</code> try to solve problems that lack polynomial-time solutions?</p> + +<p>Thankfully, there are at least two &ldquo;compromise&rdquo; alternatives.</p> + +<h3 id="forgetful-space-efficiency">Forgetful space-efficiency</h3> +<!-- "no operation relies on e being a T2, skipping the check doesn't risk soundness" p.12--> +<!-- "In forgetful \lambda_H, we offer a simple solution to space inefficient casts: just forget about them" p.11--> +<!-- "Just the same, when accumulating casts on the stack, we throw away all but the last cast" p.11--> +<!-- "forgetful ... skip[s] checks and change[s] blame labels" p.3--> + +<blockquote> + <p>&ldquo;Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 10, bottom of page 23--></blockquote> + +<p>The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:</p> + +<ol> + <li>the client obligations from the old contract, and</li> + <li>the server obligations from the new contract</li></ol> + +<p>If not, proceed as usual &mdash; by wrapping (an unwrapped value) or raising an error. Every value receives at most <strong>one</strong> wrapper; this wrapper changes as the value flows to different clients.</p> + +<p>Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module <code>A</code> exports a function <code>f</code> with contract <code>(-&gt; T1 T2)</code> to module <code>B</code>, and suppose module <code>B</code> shares this function with a few other client modules using different contracts. As <code>f</code> flows to a new client, it keeps the <code>T1</code> domain check and gets a replacement for the <code>T2</code> codomain check.</p> + +<ul> + <li>Keeping <code>T1</code> ensures that the code inside the function (defined by module <code>A</code>) receives input that matches its expectation.</li> + <li>Replacing <code>T2</code> ensures that each new client receives output that it expects.</li></ul> + +<p>Unfortunately, replacing <code>T2</code> also means that clients of module <code>B</code> cannot trust the <code>T2</code> contract. This contract is not checked, and so forgetful contracts <strong>miss</strong> some errors that would be caught by standard contracts. For the same reason, a bug in module <code>B</code> may go undetected by its clients &mdash; even if a later contract reports an issue, the contract system has no memory that <code>B</code> was partly-responsible.</p> + +<p>Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.</p> + +<h3 id="heedful-space-efficiency">Heedful space-efficiency</h3> + +<p>A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set &mdash; or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.</p> + +<p>To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.</p> + +<p>The heedful method is a compromise between forgetful and eidetic.</p> + +<ul> + <li> + <p>Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of <code>contract-stronger?</code> predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.</p></li> + <li> + <p>Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.</p></li></ul> + +<p>For details, see <a href="https://arxiv.org/abs/1410.2813">the extended version</a> of Michael&rsquo;s POPL 2015 paper. Don&rsquo;t bother searching <a href="http://www.cs.pomona.edu/~michael/papers/popl2015_space.pdf">the conference version</a> &mdash; aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.</p> + +<p><code>*</code> If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to <a href="http://mballantyne.net/">Michael Ballantyne</a> for explaining this to me.</p> + +<h3 id="priorities-and-appearances">Priorities and Appearances</h3> + +<p>The extended version of <em>Space-Efficient Manifest Contracts</em> introduces the forgetful and heedful methods with extreme modesty. It&rsquo;s tempting to skip past them and focus on the eidetic method.</p> + +<blockquote> + <p>&ldquo;Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [<a href="https://dl.acm.org/citation.cfm?doid=1706299.1706342">Siek and Wadler 2010</a>]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen&mdash;warm up exercises for eidetic.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 1, bottom of page 2--></blockquote> + +<p>And yet, at least two other research papers rely on these &ldquo;strawmen&rdquo; &mdash; or rather, the ideas behind the names.</p> + +<p><a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a>, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.</p> + +<blockquote> + <p>&ldquo;if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- page 21--></blockquote> + +<p>This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.</p> + +<blockquote> + <p>&ldquo;removing these casts preserves the soundness of the evaluation while reducing the number of them&rdquo;</p> + <p>&ldquo;while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- p.27--><!-- page 21--></blockquote> +<!-- The followup at POPL 2019 is not forgetful.--> +<!-- It's similar to eager coercions ... keep all types around and error--> +<!-- if there's a new type that doesn't match the old ones.--> +<!-- Also, that paper chooses not to let functions have intersection types,--> +<!-- which kind-of-avoids the questions ... but really the eagerness is key.--> + +<p><a href="https://dl.acm.org/citation.cfm?id=3009849"><em>Big Types in Little Runtime</em></a>, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their <em>transient</em> semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.</p> + +<p>That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value&rsquo;s set. When a mismatch occurs, the semantics <a href="https://www.ccs.neu.edu/home/types/resources/notes/transient-undefined-blame-extract.pdf">tries to report</a> every type in the set that relates to the mismatch.</p> + +<p>And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">ICFP 2018</a>. Where will we see them next?</p> + + Writing a paper with Scribble + + urn:http-prl-ccs-neu-edu:-blog-2019-02-17-writing-a-paper-with-scribble + 2019-02-17T16:20:50Z + 2019-02-17T16:20:50Z + + Ben Greenman + +<p>This post explains how to get started using Scribble to write a research paper.</p> +<!-- more--> + +<hr /> + +<blockquote> + <p>This post was written using <a href="http://download.racket-lang.org/all-versions.html">Racket 7.1</a> and <a href="https://github.com/racket/scribble/releases/tag/v7.1">Scribble 1.29</a></p></blockquote> + +<p>Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:</p> + +<ul> + <li>the paper can import definitions from the research, keeping a single point of control;</li> + <li>the language&rsquo;s functional abstractions can help manage the writing;</li> + <li>the language&rsquo;s drawing and/or plotting libraries can replace <a href="https://ctan.org/pkg/pgf?lang=en">TikZ</a>;</li> + <li>and you can write unit tests to validate the claims made in the paper.</li></ul> + +<p>Scribble, <a href="http://docs.racket-lang.org/scribble/index.html">the Racket documentation tool</a>, comes with a to-LaTeX compiler and a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> library tailored to the new <a href="https://ctan.org/pkg/acmart?lang=en">ACM paper format</a>. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short &ldquo;getting started&rdquo; guide and links to some larger examples.</p> + +<blockquote> + <p>For a Scribble tutorial, see the links in: <a href="/blog/2017/05/23/building-a-website-with-scribble/index.html">Building a Website with Scribble</a></p></blockquote> + +<h2 id="getting-started-with-">Getting started with <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a></h2> + +<p>The first line of a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> document sets the formatting options (similar to a LaTeX file using <code>acmart.cls</code>). For example, the <a href="https://conf.researchr.org/track/gpce-2018/gpce-2018#Call-for-Papers">GPCE 2018 call for papers</a> asks for anonymized <code>sigplan</code>-format submissions with line numbers and 10 point font. The proper Scribble incantation is:</p> + +<pre><code>#lang scribble/acmart @sigplan @anonymous @review @10pt</code></pre> + +<p>Next, you may want to import some definitions. If we have a file <code>references.rkt</code> (see below for a definition), we can import it as follows:</p> + +<pre><code>@require{references.rkt}</code></pre> + +<p>The third main ingredient is the title and author information:</p> + +<pre><code>@(define neu (affiliation #:institution "Northeastern University")) +@(define anon (email "anon@anon.net")) + +@title{Writing a paper with Scribble} +@author[#:affiliation neu #:email anon]{Ben Greenman} + +@; optional: set the author names in the page headers +@elem[#:style "Sshortauthors"]{B. Greenman}</code></pre> + +<p>The paper is now ready to be written. You can forge ahead with a new <a href="http://docs.racket-lang.org/scribble/base.html#%28def._%28%28lib._scribble%2Fbase..rkt%29._section%29%29">section</a> and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and <a href="http://docs.racket-lang.org/scribble/base.html#%28form._%28%28lib._scribble%2Fbase..rkt%29._include-section%29%29">import</a> content from other modules:</p> + +<pre><code>@include-abstract{abstract.scrbl} +@include-section{introduction.scrbl}</code></pre> + +<p>Finally, the main page is a good place to <a href="https://docs.racket-lang.org/scriblib/autobib.html">generate the bibliography</a>. Assuming this document imports a file like the <code>references.rkt</code> below, this expression inserts a bibliography titled &ldquo;References&rdquo;:</p> + +<pre><code>@generate-bibliography[#:sec-title "References"]</code></pre> + +<p>To build the document, invoke <code>scribble</code> on the command-line with the <code>--pdf</code> or <code>--latex</code> options:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl</code></pre> + +<p>If all goes well, this command generates a <code>FILE.pdf</code> with properly-linked cross references.</p> + +<h3 id="auxiliary-files">Auxiliary Files</h3> + +<p>If you save the code above to a file <code>example.scrbl</code> and save the files below in the same directory, then you should be able to build an <code>example.pdf</code>.</p> + +<p>These files are available in a slightly different format at this link:</p> + +<ul> + <li><a href="https://gitlab.com/bengreenman/scribble-acmart-example">https://gitlab.com/bengreenman/scribble-acmart-example</a></li></ul> + +<h4 id="referencesrkt"><code>references.rkt</code></h4> + +<pre><code>#lang racket/base + +(provide + ~cite citet generate-bibliography + fbf-icfp-2009) + +(require + scriblib/autobib) + +(define-cite ~cite citet generate-bibliography + #:style author+date-square-bracket-style) + +(define icfp "ICFP") + +(define fbf-icfp-2009 + (make-bib + #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools" + #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler") + #:location (proceedings-location icfp #:pages '(109 120)) + #:date 2017))</code></pre> + +<h4 id="abstractscrbl"><code>abstract.scrbl</code></h4> + +<pre><code>#lang scribble/acmart + +A simple Scribble document.</code></pre> + +<h4 id="introductionscrbl"><code>introduction.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require{references.rkt} + +@; start with `title` instead of `section`, because importing via +@; `include-section` shifts all title/section/subsections down one level +@title{Introduction} + +Scribble creates a connection between a stand-alone document and the artifact +it describes@~cite[fbf-icfp-2009].</code></pre> + +<h3 id="q-how-to-debug-scribble-error-messages">Q. How to debug Scribble error messages?</h3> + +<p>If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.</p> + +<p>As a compile-time example, adding <code>@ foo</code> to a document produces the message <code>unexpected whitespace after @</code> and you can either delete the whitespace or change the <code>@</code> to <code>@"@"</code> for a literal <code>@</code>-sign.</p> + +<p>As a run-time example, adding <code>@(+ 2 2)</code> produces this message:</p> + +<pre><code>not valid in document body (need a pre-part for decode) in: 4</code></pre> + +<p>One fix is to convert <code>4</code> to a string, as in <code>@~a[(+ 2 2)]</code>.</p> + +<p>But if something goes wrong when Scribble renders a generated document to PDF, the default error output is <strong>not</strong> likely to help. For example, adding <code>@elem[#:style "oops"]</code> to a document produces a giant message:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl +[[ ... 84K of output ... ]] +Output written on example.pdf (1 page, 277876 bytes). +PDF statistics: + 53 PDF objects out of 1000 (max. 8388607) + 37 compressed objects within 1 object stream + 7 named destinations out of 1000 (max. 500000) + 36877 words of extra memory for PDF output out of 42996 (max. 10000000) + +run-pdflatex: got error exit code + context...: + [[ ... 17 more lines ... ]]</code></pre> + +<p>The best way to debug these messages is to <strong>ignore them</strong> and use a LaTeX compiler directly. For the &ldquo;oops&rdquo; mistake, LaTeX stops at the undefined control sequence &mdash; giving a hint about how to find the problem:</p> + +<pre><code>$ raco scribble --latex FILE.scrbl +$ pdflatex FILE.tex +[[ ... 12KB of output ... ]] +! Undefined control sequence. +l.549 \oops + {} +? </code></pre> + +<h3 id="q-how-to-add-a-latex-style-file">Q. How to add a LaTeX style file?</h3> + +<p>To add extra LaTeX code to the final document, create a new file and include it with the <code>++style</code> command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).</p> + +<pre><code>$ raco scribble ++style style.tex --pdf FILE.scrbl</code></pre> + +<p>Here is an example style file.</p> + +<h4 id="styletex"><code>style.tex</code></h4> + +<pre><code>\settopmatter{printfolios=true,printccs=true,printacmref=true} +% add page numbers etc. + +\overfullrule=1mm +% draw a black rectangle near lines that overflow the margin</code></pre> + +<p>Another way to add extra LaTeX code is to add a <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Flatex-properties..rkt%29._tex-addition%29%29"><code>tex-addition</code></a> style property to the main title. This second approach makes it easy to include more than one file:</p> + +<pre><code>#lang scribble/acmart + +@require[ + (only-in scribble/core make-style) + (only-in scribble/latex-properties make-tex-addition)] + +@(define extra-style-files + (list (make-tex-addition "style.tex"))) + +@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble} + +@; ....</code></pre> + +<h3 id="q-how-to-make-a-figure">Q. How to make a figure?</h3> + +<p>Use the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library to add figures to a document.</p> + +<pre><code>@require[pict scriblib/figure] +@figure[ + "fig:fish" @; figure tag, see `figure-ref` + @elem{A Standard Fish} @; figure caption, appears below the content + @elem{fish = @(standard-fish 90 40)}] @; content</code></pre> + +<p>The content of a figure can be almost anything that would work in the toplevel of the document.</p> + +<h3 id="q-how-to-include-extra-files-pictures-latex">Q. How to include extra files (pictures, LaTeX)?</h3> + +<p>The <code>++extra</code> command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.</p> + +<p>For example, if a document includes the content of an external LaTeX file:</p> + +<pre><code>@elem[#:style "input"]{inline-this.tex}</code></pre> + +<p>then make sure to build the document with a command like this:</p> + +<pre><code>$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl</code></pre> + +<h4 id="inline-thistex"><code>inline-this.tex</code></h4> + +<pre><code>% Raw LaTeX allowed here +$\lambda x.\, x$</code></pre> + +<h3 id="q-what-about-in-line-latex">Q. What about in-line LaTeX?</h3> + +<p>An <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Fcore..rkt%29._element%29%29">element</a> with the <a href="https://docs.racket-lang.org/scribble/core.html#%28idx._%28gentag._60._%28lib._scribblings%2Fscribble%2Fscribble..scrbl%29%29%29"><code>'exact-chars</code></a> <a href="https://docs.racket-lang.org/scribble/core.html#%28tech._style._property%29">style property</a> renders directly to LaTeX.</p> + +<pre><code>@(define (exact . stuff) + @; the style name "relax" puts a `\relax` no-op in front of the stuff + (make-element (make-style "relax" '(exact-chars)) stuff)) + +@exact|{$\lambda x.\, x$}| +@; ==&gt; \relax{$\lambda x.\, x$} + +@(define ($ . math-stuff) + (apply exact (list "$" math-stuff "$"))) + +@${\lambda x.\, x} +@; ==&gt; \relax{$\lambda x.\, x$}</code></pre> + +<h2 id="creating-a-httpdocsracket-langorgguidemodulesyntaxhtml28parthash-lang29lang-for-a-paper">Creating a <a href="http://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29">#lang</a> for a paper</h2> + +<p>For a Scribble document that is split across multiple files, it can be helpful to make a <code>#lang</code> that <a href="http://blog.racket-lang.org/2017/03/languages-as-dotfiles.html">provides a common environment</a>. Instead of starting each file with a <code>require</code>, e.g.:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require["references.rkt" "helper-functions.rkt" scriblib/figure] + +....</code></pre> + +<p>files can start with a name that describes their common purpose:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang conference-2018-submission + +....</code></pre> + +<p>As a bonus, if the language is defined as a package then the Scribble document can use Racket&rsquo;s dependency management tools:</p> + +<pre><code># to install the paper and interactively install dependencies: +$ cd conference-2018-submission; +$ raco pkg install + +# To check that the paper builds with no dependency issues: +$ raco setup --check-pkg-deps conference-2018-submission + +# To run all unit tests +$ raco test -c conference-2018-submission</code></pre> + +<p>To create a package and language:</p> + +<ol> + <li>Move the Scribble document to a directory with the language name, i.e., <code>conference-2018-submission/</code></li> + <li>Write a simple <code>info.rkt</code> to configure the package</li> + <li>Create a normal Racket module that exports the common environment</li> + <li>Create a <code>conference-2018-submission/lang/reader.rkt</code> module</li></ol> + +<p>Details below. For a full example, visit:</p> + +<ul> + <li><a href="https://gitlab.com/bennn/scribble-acmart-example">https://gitlab.com/bennn/scribble-acmart-example</a></li></ul> + +<hr /> + +<h4 id="conference-2018-submissioninforkt"><code>conference-2018-submission/info.rkt</code></h4> + +<p>This file defines the basic metadata for a package. For more about <code>info.rkt</code>, see: <a href="http://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html">Tutorial: Creating a Package</a>.</p> + +<pre><code>#lang info +(define collection "conference-2018-submission") +(define deps '("base" "scribble-lib" "at-exp-lib")) +(define build-deps '("racket-doc" "scribble-doc")) +(define pkg-desc "Paper for Conference 2018") +(define version "0.1")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionmainrkt"><code>conference-2018-submission/main.rkt</code></h4> + +<p>This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> language, the file &ldquo;references.rkt&rdquo;, and the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library.</p> + +<pre><code>#lang racket/base + +(provide + (all-from-out + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")) + +(require + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionlangreaderrkt"><code>conference-2018-submission/lang/reader.rkt</code></h4> + +<p>This file: (1) tells Racket to use the Scribble reader on <code>#lang conference-2018-submission</code> modules, and (2) wraps the result of such modules in a shape that Scribble expects.</p> + +<pre><code>#lang s-exp scribble/base/reader +conference-2018-submission +#:wrapper1 (lambda (t) (cons 'doc (t)))</code></pre> + +<h2 id="links-to-example-documents">Links to Example Documents</h2> + +<p>These documents use the <code>#lang</code> approach to writing a paper with Scribble. Check their <code>main.rkt</code> for example formatting functions and unit tests, and check the <code>.scrbl</code> files to see how the ideas above look in a larger document.</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018">https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018</a></li> + <li><a href="https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018">https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018</a></li></ul> + +<p>Finally, this repository provides a tool to start a new Scribble document:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gtp-paper">https://pkgd.racket-lang.org/pkgn/package/gtp-paper</a></li></ul> + +<h2 id="further-reading">Further Reading</h2> + +<ul> + <li><a href="https://project.inria.fr/coqexchange/checking-machine-checked-proofs/">Checking Machine-Checked Proofs</a></li></ul> + + The Behavior of Gradual Types: A User Study + + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + 2018-12-11T19:50:33Z + 2018-12-11T19:50:33Z + + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + Java and Migratory Typing + + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + 2018-12-02T14:41:53Z + 2018-12-02T14:41:53Z + + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> + + Disappearing Code + + urn:http-prl-ccs-neu-edu:-blog-2018-11-24-disappearing-code + 2018-11-24T09:52:58Z + 2018-11-24T09:52:58Z + + Ben Greenman + +<p>Two experiences at <a href="https://2018.splashcon.org/home">SPLASH 2018</a> reminded me that software gets thrown away and replaced.</p> +<!-- more--> + +<h3 id="story-1">Story 1</h3> + +<p>The first reminder came near the end of a <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a> by <a href="https://people.csail.mit.edu/rinard/">Martin Rinard</a>. Once upon a time, Martin was working as a consultant and a firm asked him to review a software package. (The firm wanted a second opinion about how the software computed its results.) The firm sent a zipfile; Martin found six versions of the code inside; the firm said &ldquo;well, please check all six versions&rdquo;; and it turned out:</p> + +<ul> + <li><strong>Version 1</strong> : the source code was written in a domain-specific language (DSL) that generated code for the application</li> + <li><strong>Version 2</strong> : the DSL source was the same as version 1, but the generated code was slightly modified</li> + <li>&hellip;</li> + <li><strong>Version 6</strong> : the generated code was the source code and the DSL was gone</li></ul> + +<p>The moral of Martin&rsquo;s story was: (1) the creators of a software system are often different from the maintainers, and (2) researchers need to build tools to help these maintainers.</p> + +<h3 id="story-2">Story 2</h3> + +<p>The second reminder came from a teaching assistant who said the <a href="https://www.cs.cornell.edu/courses/cs3110/2018fa/">functional programming course</a> at their institution was currently using a Python script to test students&rsquo; code. Once upon a time, I was a teaching assistant for the <a href="https://www.cs.cornell.edu/courses/cs3110/2014sp/">same course</a> at the same institution. We had trouble testing students&rsquo; code via the Python script left by the pre&ndash;2013 course staff, so I wrote a <a href="https://gitlab.com/bengreenman/ocaml_tools/">command-line tool</a> to handle the tests and other compile/run/grade tasks. To keep history from repeating itself, I used the same language the course teaches (OCaml) and wrote some documentation &mdash; but it seems like that was not enough. At any rate, writing the tool was a good exercise.</p> + +<blockquote> + <p><em>In the end, everybody must understand for himself.</em> &mdash; <a href="https://dl.acm.org/citation.cfm?id=3731">Per Martin-Löf</a></p></blockquote> + +<h3 id="reflection">Reflection</h3> + +<p>In each story, the maintainers of a software system threw away some old code to make their job easier in the short term. How can we stop this &ldquo;re-inventing the wheel&rdquo; from happening?</p> + +<p>Martin Rinard&rsquo;s solution is to let maintenance programmers keep their current habits, but provide tools to make the short-term, pragmatic solutions into a more robust systems. Search for "<a href="https://people.csail.mit.edu/rinard/paper/osdi04.pdf">failure-oblivious computing</a>" to learn more (this was the topic of his <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a>).</p> + +<p>In Story 1, the maintainers were able to avoid the DSL by modifying an inherited blob of DSL-generated code. If the DSL did not generate code, history might have taken a different course; it might be best to start with a language that offers tools for linguistic re-use, and to build a DSL from these tools &mdash; so there is no generated code. The Racket programming language is exploring this path. For a recent example, see the <a href="https://www2.ccs.neu.edu/racket/pubs/icfp17-acf.pdf">video-lang paper</a>.</p> + +<p>The Story 2 test harness, however, was not generating code. Its maintainers discarded a &ldquo;big&rdquo; program written in a typed functional language in favor of a script. Perhaps we need a language that allows mixing statically-typed and dynamically-typed code (shouts out to <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">my own research</a>).</p> + +<p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p> + +<hr /> + +<h4 id="addendum-comment-from-mitch-wand">Addendum: comment from Mitch Wand</h4> + +<blockquote> + <p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p></blockquote> + +<p>Ermm, this works better for sourdough bread than for people.</p> + +<p>Even in the not-so-real world of checking student solutions, there&rsquo;s often no way of guaranteeing that one half of a pair will be around for the second round. They may be on co-op. Or the course will not be offered the next semster/year/etc. Or the course will change at the next offering (from OCaml to Python or from Racket to Java) so that large chunks of the infrastructure will have to be discarded or rewritten.</p> + +<p>The &ldquo;real&rdquo; solution is to write literate code (as we preached incessantly in PDP), so that the next reader will have at least some clue as about what you wrote. This just may be sufficient incentive to modify rather than rebuild from scratch.</p> + +<p>Ever the optimist, &mdash;Mitch</p> + + A Spectrum of Type Soundness and Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-10-06-a-spectrum-of-type-soundness-and-performance + 2018-10-06T11:23:35Z + 2018-10-06T11:23:35Z + + Ben Greenman + +<p>The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>A Spectrum of Type Soundness and Performance</em> by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018">http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018</a></p></blockquote> + +<p>A dynamically-typed language runs any program that &ldquo;looks good&rdquo; (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both &ldquo;looks good&rdquo; and is well-typed according to a type checker.</p> + +<p>A <em>mixed-typed</em> language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.</p> + +<div class="figure"><img src="/img/mixed-typed-systems-by-year.png" alt="Figure 1: Some mixed-typed languages" /> + <p class="caption">Figure 1: Some mixed-typed languages</p></div> + +<p>These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:</p> + +<ul> + <li><strong>MACLISP</strong> defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the <a href="http://www.softwarepreservation.org/projects/LISP/MIT/Moon-MACLISP_Reference_Manual-Apr_08_1974.pdf">Moonual</a>. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).</li> + <li><strong>Strongtalk</strong> includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a> position paper.</li> + <li><strong>Typed Racket</strong> lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.</li> + <li><strong>Thorn</strong> requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.</li> + <li><strong>Reticulated</strong> lets a program combine static and dynamic <em>expressions</em> and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek <a href="https://dl.acm.org/citation.cfm?id=3009849">POPL 2017</a>).</li></ul> + +<p>That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system&rsquo;s protocol for checking dynamically-typed values at a <em>type boundary</em> (between static and dynamic code).</p> + +<h3 id="main-contribution">Main Contribution</h3> + +<p>In the paper <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>A Spectrum of Type Soundness and Performance</em></a>, we define a tiny mixed-typed language and show three ways to define the behavior of programs &mdash; based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.</p> + +<p>The three behaviors are inspired by existing languages. A <strong>higher order</strong> behavior ensures that dynamically-typed values match the static type at a boundary &mdash; by checking the value when possible, and by monitoring the value&rsquo;s future interactions when necessary. A <strong>first order</strong> behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An <strong>erasure</strong> behavior does no checking whatsoever.</p> + +<blockquote> + <p>Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function <code>f</code>, then one way to enforce the type boundary is to wrap <code>f</code> in a proxy to assert that every future call to <code>f</code> returns a number. In this case, the proxy monitors the behavior of <code>f</code>.</p></blockquote> + +<p>Concretely, the paper defines three formal semantics with the same names. The <strong>higher-order</strong> semantics enforces full types at the boundaries (Section 2.3). The <strong>first-order</strong> semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every &ldquo;selector&rdquo; operation in typed code, e.g., function application, data structure access (Section 2.5). The <strong>erasure</strong> semantics simply ignores the types (Section 2.4).</p> + +<p>Each semantics satisfies a <em>different</em> notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">online supplement</a> gives full proofs.</p> + +<p>The paper has more to say about the meta-theory. See section 2 and section 4.</p> + +<blockquote> + <p>To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for <a href="https://dl.acm.org/citation.cfm?id=2676971">subset of the language</a> (in the spirit of <a href="http://soundiness.org/">soundiness</a>) or use the name &ldquo;type soundness&rdquo; to describe <a href="https://dl.acm.org/citation.cfm?id=2676971">a different property</a>.</p></blockquote> + +<p>Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations &mdash; similar to Typed Racket&rsquo;s <a href="http://docs.racket-lang.org/ts-reference/Typed_Racket_Syntax_Without_Type_Checking.html">no-check</a> language.</p> + +<p>Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (<a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016</a>). The programs we measured are the non-object-oriented ones from our <a href="http://docs.racket-lang.org/gtp-benchmarks/index.html">benchmark suite</a>.</p> + +<p>To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises &mdash; see section 3 of the paper, section B of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, and/or the <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slides</a>.</p> + +<h3 id="implications">Implications</h3> + +<ol> + <li>The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, or <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slide 114</a>.</li> + <li>Programmers using mixed-typed languages need to know what guarantees their types provide. (It is <a href="https://twitter.com/jbandi/status/965005464638541825">not safe to assume that TypeScript types give the same guarantees as OCaml types</a>!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.</li> + <li>The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?</li></ol> + +<blockquote> + <p>Note: in this post, a <em>mixed-typed language</em> is one that allows any combination of static and dynamic typing. A <em>gradually-typed language</em> is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>).</p></blockquote> + + Sampling Gradual Typing Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + 2018-05-08T15:37:37Z + 2018-05-08T15:37:37Z + Ben Greenman + Zeina Migeed + + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> + + The Racket School 2018: Create your own language + + urn:http-prl-ccs-neu-edu:-blog-2018-04-27-the-racket-school-2018-create-your-own-language + 2018-04-27T21:35:22Z + 2018-04-27T21:35:22Z + + Ben Greenman + +<p>The Racket School 2018: Create your own language • 9–13 July • Salt Lake City</p> +<!-- more--> + +<p>The Racket team has spent over thirty years developing and refining a coherent intellectual tradition for studying and building programming languages. This year’s school will introduce participants to Racket’s framework for language-oriented programming, which the summer school faculty recently spelled out in a a cover article in the <a href="https://tinyurl.com/RacketCACM">Communications of the ACM</a></p> + +<p>Concretely, the 2018 Racket Summer School will cover the following topics:</p> + +<ul> + <li>the spectrum of programming languages;</li> + <li>modules and syntax, or languages as libraries;</li> + <li>DrRacket’s support for language-oriented programming;</li> + <li>a domain-specific language for adding types to languages;</li> + <li>tools and techniques for implementing notational conveniences; and</li> + <li>research challenges in language-oriented programming.</li></ul> + +<p>If these topics intrigue you, attend the Racket Summer School:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2018/">http://summer-school.racket-lang.org/2018/</a></li></ul> + +<p>This is not your run-of-the-mill summer school. We will do our best to make it exciting, entertaining, and useful to a broad spectrum of attendees, both academic and industrial.</p> + +<p>P.S. We will send you your first problem set in June, a month before the summer school to whet your appetite.</p> + + PLT Redex FAQ + + urn:http-prl-ccs-neu-edu:-blog-2017-09-25-plt-redex-faq + 2017-09-25T23:39:16Z + 2017-09-25T23:39:16Z + Ben Greenman + Sam Caldwell + + Ben Greenman, Sam Caldwell + +<p>A short guide to Redex concepts, conventions, and common mistakes.</p> +<!-- more--> + +<hr /> + +<p><em>To contribute to this FAQ, submit issues and pull requests to: <a href="https://github.com/nuprl/website/">https://github.com/nuprl/website/</a></em></p> + +<h3 id="q-what-is-redex-useful-for">Q. What is Redex useful for?</h3> + +<ol> + <li>declaring <a href="https://en.wikipedia.org/wiki/Regular_tree_grammar">regular tree grammars</a></li> + <li>defining <em>pattern</em>-based judgments and relations on <em>terms</em></li> + <li>testing properties of the above</li></ol> + +<p>More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.</p> + +<h3 id="q-what-is-redex-not-useful-for">Q. What is Redex <strong>not</strong> useful for?</h3> + +<p>Proving theorems about a grammar, judgment, or relation.</p> + +<h3 id="q-what-is-a-term">Q. What is a <em>term</em>?</h3> + +<p>Informally, a term is:</p> + +<ul> + <li>a Redex &ldquo;atom&rdquo;, or</li> + <li>an object that represents a sequence of characters.</li></ul> + +<p>More formally, a term is the result of evaluating <strong>(term X)</strong>, where <strong>X</strong> is any syntactically-correct Racket expression.</p> + +<p>Examples:</p> + +<pre><code>$ racket +Welcome to Racket v6.10.0.3. +&gt; (require redex/reduction-semantics) +&gt; (term 42) +42 +&gt; (term (+ 2 2)) +'(+ 2 2) +&gt; (term ("hello" world (#false))) +'("hello" world (#f))</code></pre> + +<p>Some terms may look strange. That&rsquo;s OK, because a term by itself has no meaning.</p> + +<p>Terms can refer to previously-defined values using the <strong>unquote</strong> escape.</p> + +<pre><code>&gt; (define x (term 42)) +&gt; (term (+ 2 x)) +'(+ 2 x) +&gt; (term (+ ,x (unquote x))) +'(+ 42 42)</code></pre> + +<h3 id="q-what-is-a-redex-model">Q. What is a <em>Redex model</em>?</h3> + +<p>A Redex model is collection of tools for working with terms. The tools may include:</p> + +<ul> + <li><em>languages</em>, to define a grammar for terms</li> + <li><em>judgments</em>, to describe properties of terms or relations between terms</li> + <li><em>metafunctions</em>, to transform terms</li> + <li><em>reduction relations</em>, to define a term rewriting system</li></ul> + +<p>The goal of these tools is to encode a &ldquo;real thing&rdquo; (maybe, a programming language) using Redex terms.</p> + +<h3 id="q-what-is-a-language">Q. What is a language?</h3> + +<p>A Redex <em>language</em> is a named set of non-terminals, <em>patterns</em>, and <em>binding forms</em>. For example, a Redex model of the natural numbers might start with this language:</p> + +<pre><code>(define-language nat + [N ::= Zero + (Plus1 N)])</code></pre> + +<ul> + <li>the name of the language is <strong>nat</strong></li> + <li>the non-terminal <strong>N</strong> is associated with two patterns: <strong>Zero</strong> and <strong>(Plus1 N)</strong></li> + <li>there are no <em>binding forms</em></li></ul> + +<p>Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.</p> + +<p>The non-terminal <strong>N</strong> describes all terms that are either:</p> + +<ol> + <li>the symbol <strong>Zero</strong></li> + <li>lists of the form <strong>(Plus1 N)</strong>, where <strong>N</strong> is either <strong>Zero</strong> or another &ldquo;Plus1&rdquo;</li></ol> + +<p>For example,</p> + +<pre><code>(term Zero) +(term (Plus1 Zero)) +(term (Plus1 (Plus1 Zero))) +(term (Plus1 (Plus1 (Plus1 Zero)))) +;; .... and so on</code></pre> + +<p>If a language has binding forms, then some terms can introduce names. See the question on <em>binding forms</em> (below) for an example.</p> + +<h3 id="q-what-is-a-pattern">Q. What is a pattern?</h3> + +<p>A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains <em>named non-terminals</em> from the language, then you can ask whether a Redex term matches the pattern.</p> + +<p>A <em>named non-terminal</em> for a language <strong>L</strong> is an identifier made of: (1) a non-terminal from <strong>L</strong>, (2) an underscore (<strong>_</strong>), and (3) any other identifier. See the FAQ entry below.</p> + +<p>For example, <strong>(redex-match? L p t)</strong> returns <strong>#true</strong> if the term <strong>t</strong> matches the pattern <strong>p</strong> relative to the language <strong>L</strong>.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(redex-match? nat N_some-name (term Zero)) +;; #true +(redex-match? nat (Plus1 N_a) (term Zero)) +;; #false +(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; #true</code></pre> + +<p>If <strong>(redex-match? L p t)</strong> is <strong>#true</strong>, then <strong>(redex-match L p t)</strong> shows how named non-terminals in the pattern bind to subterms of <strong>t</strong>.</p> + +<pre><code>(redex-match nat N_0 (term Zero)) +;; (list (match (list (bind 'N_0 'Zero)))) +(redex-match nat (Plus1 N_0) (term Zero)) +;; #f +(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))</code></pre> + +<h3 id="q-what-is-a-named-non-terminal">Q. What is a named non-terminal?</h3> + +<p>A named non-terminal in a language <strong>L</strong> is an identifier of the form <strong>X_Y</strong>, where:</p> + +<ul> + <li><strong>X</strong> is a non-terminal from <strong>L</strong></li> + <li><strong>Y</strong> is any identifier</li></ul> + +<p>The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.</p> + +<pre><code>(define-language trees + [binary-tree ::= Leaf + (Node binary-tree binary-tree)]) + +(redex-match trees + (Node binary-tree_left binary-tree_right) + (term (Node Leaf (Node Leaf Leaf)))) +;; (list +;; (match +;; (list (bind 'binary-tree_left 'Leaf) +;; (bind 'binary-tree_right '(Node Leaf Leaf)))))</code></pre> + +<h3 id="q-what-else-can-patterns-express">Q. What else can patterns express?</h3> + +<p>Redex patterns may contain special identifiers to guide pattern-matching. For instance:</p> + +<ul> + <li>The <strong>_</strong> pattern matches any term (and does not bind).</li> + <li>A pattern <strong>(p &hellip;)</strong> matches any sequence whose elements match the pattern <strong>p</strong>. If the pattern <strong>p</strong> is a named non-terminal, then the non-terminal binds a sequence of subterms.</li></ul> + +<p>Examples:</p> + +<pre><code>(redex-match? nat (Plus1 _) (term (Plus1 9))) +;; #true +(redex-match? nat (N_0 ...) (term ())) +;; #true +(redex-match? nat (N_0 ...) (term (Zero))) +;; #true +(redex-match nat (N_0 ...) (term (Zero Zero Zero))) +;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))</code></pre> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29">the Redex reference</a> for the full pattern language, including the <em>named and unique non-terminals</em> of the form <strong>X_!_Y</strong>.</p> + +<h3 id="q-what-can-patterns-not-express">Q. What can patterns <strong>not</strong> express?</h3> + +<ul> + <li>Disjunctions of patterns, e.g., &ldquo;number or boolean&rdquo;. (Use a language non-terminal.)</li> + <li>Negations of patterns. (Compose <strong>not</strong> with <strong>redex-match?</strong>.)</li> + <li>Some non-regular patterns. (Named dots <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29"><code>..._N</code></a> or <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-language%29%29"><code>define-language</code></a> with a side condition may be able to help.)</li></ul> + +<h3 id="q-what-is-a-judgment">Q. What is a judgment?</h3> + +<p>A Redex <em>judgment</em> form defines a relation on terms. The relation is defined by a set of inference rules.</p> + +<p>Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:</p> + +<ul> + <li>well-formedness conditions (i.e., whether a term contains free variables)</li> + <li>type checking rules</li> + <li>type inference rules</li> + <li>evaluation relations</li></ul> + +<p>Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(define-judgment-form nat + #:mode (N= I I) + #:contract (N= N N) + [ + --- Zero= + (N= Zero Zero)] + [ + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1) + (N= N_0-- N_1--) + --- Plus1= + (N= N_0 N_1)])</code></pre> + +<ol> + <li>the language is <strong>nat</strong>; Redex uses the language to interpret patterns</li> + <li>the mode is <strong>(N= I I)</strong>; this means <strong>N=</strong> is the name of a judgment that expects two input terms (or, <strong>N=</strong> is a binary relation on terms)</li> + <li>the contract is <strong>(N= N N)</strong>; in other words, <strong>N=</strong> expects two terms that match the <strong>N</strong> non-terminal from the <strong>nat</strong> language</li> + <li>there are two inference rules, named <strong>Zero=</strong> and <strong>Plus1=</strong></li> + <li>the <strong>Zero=</strong> rule states that <strong>(N= Zero Zero)</strong> always holds</li> + <li>the <strong>Plus1=</strong> rule states that <strong>(N= N_0 N_1)</strong> holds if <strong>N_0</strong> and <strong>N_1</strong> are both <strong>Plus1</strong> terms whose contents are related by <strong>N=</strong></li></ol> + +<p>The <strong>where</strong> clauses are <em>guards</em>. When Redex tries to apply a rule with premises of the form <strong>(where pattern term)</strong>, it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-judgment-form%29%29">the Redex reference</a> for more.</p> + +<pre><code>(judgment-holds (N= Zero Zero)) +;; #true +(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Note: the inference rules form a <em>set</em>, not a <em>sequence</em>. So when you ask Redex whether <strong>(judgment-holds (N= Zero Zero))</strong>, it applies all rules that match <strong>(N= Zero Zero)</strong>. For <strong>N=</strong> this is just one rule, but in general it could be many rules.</p> + +<h3 id="q-what-is-a-judgment-form-mode">Q. What is a judgment form <strong>#:mode</strong>?</h3> + +<p>A <strong>#:mode</strong> declaration expects a list of the form <strong>(id pos-use &hellip;)</strong>, where <strong>id</strong> is an identifier and each <strong>pos-use</strong> is either <strong>I</strong> or <strong>O</strong>. These declarations say four things:</p> + +<ol> + <li><strong>id</strong> is the name of a new judgment form</li> + <li><strong>id</strong> expects <strong>N</strong> arguments, where <strong>N</strong> is the number of <strong>pos-use</strong> symbols</li> + <li><strong>id</strong> expects an <em>input</em> at each position where the mode contains an <strong>I</strong></li> + <li><strong>id</strong> produces an <em>output</em> at each position where the mode contains an <strong>O</strong></li></ol> + +<p>For example, a type inference judgment may take an expression as input and output a type. Here&rsquo;s a fast and easy type inference judgment for arithmetic expressions. Given any term <strong>e_0</strong>, the judgment outputs the type <strong>Int</strong>.</p> + +<pre><code>(define-language Arith + (e ::= integer (e + e)) + (τ ::= Int)) + +(define-judgment-form Arith + #:mode (infer-type I O) + #:contract (infer-type e τ) + [ + --- T-Int + (infer-type e_0 Int)])</code></pre> + +<h3 id="q-what-can-judgments-not-express">Q. What can judgments <strong>not</strong> express?</h3> + +<p>Redex does not support inference rules that require guessing.</p> + +<p>One example of this is a transitivity rule: "<strong>τ_0</strong> is related to <strong>τ_2</strong> if there exists a <strong>τ_1</strong> such that <strong>τ_0</strong> is related to <strong>τ_1</strong> and <strong>τ_1</strong> is related to <strong>τ_2</strong>". The following example tries to define a transitive subtyping (<strong>&lt;:</strong>) relation, but Redex rejects the <strong>S-Trans</strong> rule.</p> + +<pre><code>(define-language SomeTypes + (τ ::= (→ τ τ) Integer)) + +(define-judgment-form SomeTypes + #:mode (&lt;: I I) + #:contract (&lt;: τ τ) + [ + (&lt;: τ_0 τ_1) + (&lt;: τ_1 τ_2) + --- S-Trans + (&lt;: τ_0 τ_2)] + [ + --- S-Refl + (&lt;: τ_0 τ_0)] + [ + (&lt;: τ_dom-1 τ_dom-0) + (&lt;: τ_cod-0 τ_cod-1) + --- S-Arrow + (&lt;: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])</code></pre> + +<p>The error is that in the rule <strong>S-Trans</strong>, the named non-terminal <strong>τ_1</strong> appears in an input position but is not bound to a term.</p> + +<h3 id="q-what-is-a-metafunction">Q. What is a metafunction?</h3> + +<p>A metafunction is a term-level function on terms.</p> + +<p>Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.</p> + +<p>Here is a metafunction that returns <strong>#true</strong> when given two equal natural numbers. The definition is similar to the <strong>N=</strong> judgment form.</p> + +<pre><code>(define-metafunction nat + N=? : N N -&gt; boolean + [(N=? Zero Zero) + #true] + [(N=? N_0 N_1) + (N=? N_0-- N_1--) + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1)] + [(N=? N_0 N_1) + #false])</code></pre> + +<ul> + <li>the metafunction is named <strong>N=?</strong></li> + <li>its contract is <strong>N N -&gt; boolean</strong>, this means <strong>N=?</strong> expects 2 terms that match the <strong>N</strong> pattern and returns a term that matches the pattern <strong>boolean</strong></li> + <li>there are three cases; the second case is guarded by two <strong>where</strong> clauses</li></ul> + +<p>Any occurrence of <strong>(N=? &hellip;.)</strong> in any term is evaluated.</p> + +<pre><code>(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(term ((N=? Zero Zero) Zero)) +;; '(#true Zero) +(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Any occurrence of <strong>N=?</strong> outside a <strong>term</strong> is an error.</p> + +<p>Metafunction <strong>where</strong>-clauses are analogous to judgment form <strong>where</strong>-clauses. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._metafunction%29">the Redex reference</a> for more.</p> + +<p>Note: the cases in a metafunction form a <em>sequence</em>, not a <em>set</em>. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.</p> + +<h3 id="q-should-i-use-a-metafunction-or-a-judgment-form">Q. Should I use a metafunction or a judgment form?</h3> + +<p>Use a judgment form.</p> + +<p>Metafunctions are handy, but judgments are easier to read and debug and maintain.</p> + +<h3 id="q-what-is-a-reduction-relation">Q. What is a reduction relation?</h3> + +<p>A reduction relation is a set of term-rewriting rules.</p> + +<p>Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.</p> + +<ul> + <li>The language tells Redex how to interpret patterns.</li> + <li>The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.</li> + <li>The rewrite rules have the form <strong>(&mdash;&gt; term term guard &hellip;)</strong>. The term on the left is the input, the term on the right is the output.</li></ul> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._reduction-relation%29%29">the Redex reference</a> for a full description of the guards.</p> + +<p>The preferred way to define a reduction relation is to define a language that includes three non-terminals:</p> + +<ol> + <li>a non-terminal for the domain of the reduction relation</li> + <li>a non-terminal for a <em>subset</em> of the domain that cannot reduce further</li> + <li>a non-terminal for evaluation contexts</li></ol> + +<p>An evaluation context is a term that contains a <strong>hole</strong>. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite &mdash; in particular, the sub-term that matches the <strong>hole</strong>.</p> + +<p>In the following example, <strong>bexp</strong> is the domain of a reduction relation. A <strong>bexp</strong> term represents a boolean expression, which can be <strong>#true</strong> or <strong>#false</strong> or a conjunction of expressions or a disjunction of expressions. The boolean expressions <strong>#true</strong> and <strong>#false</strong> are also values (<strong>val</strong>); these cannot reduce further. The non-terminal <strong>E</strong> is for evaluation contexts.</p> + +<pre><code>(define-language Bool + (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp)) + (val ::= #true #false) + (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E))) + +(define step + (reduction-relation Bool + #:domain bexp + [--&gt; (in-hole E (val_lhs ∧ val_rhs)) + (in-hole E val_new) + ∧-step + (where val_new ,(and (term val_lhs) (term val_rhs)))] + [--&gt; (in-hole E (val_lhs ∨ val_rhs)) + (in-hole E val_new) + ∨-step + (where val_new ,(or (term val_lhs) (term val_rhs)))]))</code></pre> + +<p>The function <strong>apply-reduction-relation</strong> applies a reduction relation to a term and returns a list of ways that the term can step.</p> + +<pre><code>(apply-reduction-relation step (term #true)) +;; '() +(apply-reduction-relation step (term (#true ∧ #true))) +;; '(#true) +(apply-reduction-relation step (term (#true ∧ #false))) +;; '(#false) +(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true))) +;; '((#true ∧ #true))</code></pre> + +<p>Three things about the reduction relation <strong>step</strong>:</p> + +<ol> + <li>Using <strong>in-hole</strong> on the first argument of <strong>&mdash;&gt;</strong> searches a term for a subterm that Redex can apply a reduction rule to.</li> + <li>Using <strong>in-hole</strong> on the second argument of <strong>&mdash;&gt;</strong> puts a new value back into the <strong>hole</strong> in the evaluation context.</li> + <li>The unquote operator (<strong>,</strong>) escapes to &ldquo;Racket mode&rdquo; (see below) to evaluate a conjunction or disjunction.</li></ol> + +<p>A judgment or metafunction is a formal alternative to &ldquo;escaping to Racket&rdquo;, but escaping can be convenient.</p> + +<p>Note: the cases in a reduction relation form a <em>set</em>, not a <em>sequence</em>. If more than one case matches, Redex applies them all.</p> + +<h3 id="q-what-is-racket-mode-what-is-redex-mode">Q. What is &ldquo;Racket mode&rdquo;? What is &ldquo;Redex mode&rdquo;?</h3> + +<p>Code in a Redex model is sometimes evaluated in &ldquo;Racket mode&rdquo; and sometimes evaluated in &ldquo;Redex mode&rdquo;. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.</p> + +<p>Key points:</p> + +<ol> + <li>A Redex program starts in Racket mode.</li> + <li>The <strong>X</strong> in <strong>(term X)</strong> is evaluated in Redex mode &hellip;</li> + <li>&hellip; unless <strong>X</strong> contains unquoted sub-expressions. Unquoting escapes to Racket mode &hellip;</li> + <li>&hellip; and <strong>term</strong>s inside an unquoted sub-expression are evaluated in Redex mode.</li></ol> + +<p>In other words, <strong>term</strong> enters Redex mode and <strong>unquote</strong> (<strong>,</strong>) escapes back to Racket.</p> + +<p>Redex implicitly switches to Redex mode in a few other places, for example:</p> + +<ul> + <li>the right-side of a <strong>where</strong> clause is in Redex mode</li> + <li>the result of a metafunction is in Redex mode</li></ul> + +<p>When in doubt, try using an <strong>unquote</strong>. Redex will raise an exception if it finds an unquote in Racket mode.</p> + +<h3 id="q-are-side-conditions-evaluated-in-racket-mode-or-redex-mode">Q. Are <strong>side-condition</strong>s evaluated in &ldquo;Racket mode&rdquo; or &ldquo;Redex mode&rdquo;?</h3> + +<p>A <strong>(side-condition e)</strong> sometimes evaluates <strong>e</strong> as a Racket expression and sometimes evaluates <strong>e</strong> as a Redex expression.</p> + +<ul> + <li>reduction relations and metafunctions expect a <strong>Racket</strong> expression</li> + <li>judgments expect a <strong>Redex</strong> expression</li></ul> + +<h3 id="q-what-is-a-binding-form">Q. What is a binding form?</h3> + +<p>In the lambda calculus, <strong>λ</strong>-terms bind variables. A term <strong>(λ x M)</strong> means that any free occurrence of <strong>x</strong> in the sub-term <strong>M</strong> refers to the <strong>x</strong> from the <strong>λ</strong>-term.</p> + +<p>Redex can express this idea with a binding form.</p> + +<pre><code>(define-language Λ + [e ::= (e e) x (λ x e)] + [x ::= variable-not-otherwise-mentioned] + #:binding-forms + (λ x_0 e_0 #:refers-to x_0))</code></pre> + +<p>Note: all the non-terminals in a language must be defined before the <strong>#:binding-forms</strong> keyword. If a non-terminal definition appears after the <strong>#:binding-forms</strong> keyword, then Redex will interpret the &ldquo;definition&rdquo; as a binding form.</p> + +<p>Binding forms work together with Redex&rsquo;s functions for substitution and alphabetic equivalence.</p> + +<pre><code>(alpha-equivalent? Λ + (term (λ x x)) + (term (λ y y)))) +;; #true + +(define-metafunction Λ + test-substitute : e -&gt; e + [(test-substitute (λ x_0 e_0)) + (substitute e_0 x_0 y)]) +(term (test-substitute (λ z (z z)))) +;; '(y y)</code></pre> + +<h3 id="q-what-is--what-is-">Q. What is <strong>&hellip;</strong>? What is <strong>&hellip;.</strong>?</h3> + +<p>Three dots (<strong>&hellip;</strong>) is for building patterns. If <strong>p</strong> is a pattern then <strong>(p &hellip;)</strong> matches any list whose elements all match <strong>p</strong>.</p> + +<pre><code>(define-language L) +(redex-match? L (number ... boolean ...) (term (1 2 #true #true))) +;; #true</code></pre> + +<p>Four dots (<strong>&hellip;.</strong>) may be used in <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-extended-language%29%29"><strong>define-extended-language</strong></a> to extend a previously-defined non-terminal.</p> + +<pre><code>(define-language C + (keyword ::= auto break case)) +(define-extended-language C++ + C + (keyword ::= .... class)) + +(redex-match? C keyword (term auto)) +;; #true +(redex-match? C keyword (term class)) +;; #false +(redex-match? C++ keyword (term auto)) +;; #true +(redex-match? C++ keyword (term class)) +;; #true</code></pre> + +<h3 id="q-where-to-learn-more-about-redex">Q. Where to learn more about Redex?</h3> + +<p>&ldquo;Critical path&rdquo; resources:</p> + +<ul> + <li>Code examples for this post: <a href="https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt">https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt</a></li> + <li>Redex documentation: <a href="http://docs.racket-lang.org/redex/index.html">http://docs.racket-lang.org/redex/index.html</a></li> + <li><a href="https://dvanhorn.github.io/redex-aam-tutorial/"><em>An Introduction to Redex with Abstracting Abstract Machines</em></a> by David Van Horn</li> + <li><a href="https://www.leafac.com/playing-the-game-with-plt-redex/"><em>Playing the Game with PLT Redex</em></a> by Leandro Facchinetti</li> + <li><a href="https://williamjbowman.com/doc/experimenting-with-redex/index.html"><em>Experimenting with Languages in Redex</em></a> by William J. Bowman</li></ul> + +<p>&ldquo;Procrastination&rdquo; resources:</p> + +<ul> + <li><a href="http://tata.gforge.inria.fr/"><em>Tree Automata Techniques and Applications</em></a></li> + <li><a href="http://lamport.azurewebsites.net/pubs/lamport-types.pdf"><em>Should your Specification Language be Typed?</em></a></li> + <li>Redex source code (see <code>redex-lib/</code>): <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + + Gradual Typing Across the Spectrum, part II + + urn:http-prl-ccs-neu-edu:-blog-2017-08-22-gradual-typing-across-the-spectrum-part-ii + 2017-08-22T15:54:06Z + 2017-08-22T15:54:06Z + + Ben Greenman + +<p>Last week, Northeastern hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:</p> + +<p><a href="http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html">http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html</a></p> + +<p>A common thread among the talks was the question: <em>how to convert a research idea into a tool for software developers?</em></p> +<!-- more--> + +<p>In my mind, gradual typing <em>is</em> an answer to one instance of this question. The research idea is strong static type systems, and the software developers are the millions using dynamically typed languages. I know that static typing can make programs easier to write and maintain. The developers know that dynamic typing has benefits; moreover they know better than to migrate their code from one language to another on a whim. Gradual typing is a linguistic solution to the problem of <em>adding</em> the benefits of static typing to a dynamically typed language.</p> + +<p>Enough opinions, let&rsquo;s talk about the talks.</p> + +<p>The morning session consisted of four talks:</p> + +<ul> + <li> + <p><a href="https://www.cs.umd.edu/people/milod">Milod Kazerounian</a> (<a href="https://www.cs.umd.edu/">UMD</a>) spoke about upgrading the <a href="https://github.com/plum-umd/rdl">RDL</a> type checker for Ruby with support for refinement types. The idea is to compile Ruby code and types to <a href="https://emina.github.io/rosette/">Rosette</a>, and profit from <a href="http://yices.csl.sri.com/papers/cav2007.pdf">SMT</a>-assisted type checking.</p></li> + <li> + <p><a href="http://ambrosebs.com/">Ambrose Bonnaire-Sergeant</a> (<a href="https://www.cs.indiana.edu/">IU</a>, <a href="http://ambrosebs.com/talks/squash-work-boston-pi-2017.pdf">slides</a>) has been inferring <em>useful</em> <a href="http://typedclojure.org/">Typed Clojure</a> types through dynamic analysis of Clojure programs. His tool observes how values flow through a program at run-time, then lifts these observations into possibly-recursive, possibly-incorrect type annotations. The surprising result is that the tool quickly (1&ndash;2 seconds per unit test, I think) infers types that can help a developer start annotating a program.</p></li> + <li> + <p><a href="http://ccs.neu.edu/~types/">Ben Greenman</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>, <a href="http://homedirs.ccs.neu.edu/types/resources/talks/preservation-types.pdf">slides</a>) explained why he is implementing a semantics for <a href="https://github.com/racket/typed-racket">Typed Racket</a> inspired by Michael Vitousek&rsquo;s work on <a href="http://homes.soic.indiana.edu/mvitouse/papers/popl17.pdf">Reticulated Python</a>. The &ldquo;why&rdquo; is &ldquo;performance&rdquo;. The Reticulated semantics will enforce a notion of tag soundness in kind of <a href="https://en.wikipedia.org/wiki/Deal_with_the_Devil">devils contract</a> to improve performance.</p></li> + <li> + <p><a href="https://cs.brown.edu/~ptunnell/">Preston Tunnell-Wilson</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tpk-crowdsource-lang-design/">ONWARD 2017</a>) recently sent questions about programming language design to <a href="https://www.mturk.com/mturk/welcome">Mechanical Turk</a> workers. Survey says, developers have extremely diverse opinions about what they <em>expect</em> and what they <em>want</em> regarding scope, inheritance, and infix operators.</p></li></ul> + +<p>In the early afternoon, we had two talks on similar themes as the morning session:</p> + +<ul> + <li> + <p><a href="https://github.com/akuhlens">Andre Kuhlenschmidt</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is exploring the design space of efficient implementations for run-time type checks. The main challenge is how to <em>monitor</em> higher-order data in a way that efficiently performs type checks and can help the programmer debug any failed checks. This talk presented data comparing two approaches to the program; I believe the latter, improved approach is based on <a href="http://homepages.inf.ed.ac.uk/wadler/papers/coercions/coercions.pdf">coercions</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) explained that there are many ways to adapt type soundness to a gradually typed language, and presented some data comparing Typed Racket&rsquo;s <em>generalized soudness</em> to Reticulated Python&rsquo;s <em>tag soundness</em>. The data suggests that tag soundness never adds an order-of-magnitude slowdown.</p></li></ul> + +<p>Next on the schedule were two talks about implementing advanced type systems in Racket&rsquo;s macro expander (think: meta-level linguistic re-use, capture-avoiding substitution for free!)</p> + +<ul> + <li> + <p><a href="https://github.com/iitalics">Milo Turner</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) first showed how to implement <a href="https://gankro.github.io/blah/linear-rust/#definitions-and-the-state-of-rust">linear and affine</a> type systems using <a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html">syntax-parse</a>, and second presented a simpler implementation using the <a href="http://docs.racket-lang.org/turnstile/index.html">Turnstile</a> library.</p></li> + <li> + <p><a href="http://www.davidchristiansen.dk/">David Christiansen</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is building <a href="https://github.com/david-christiansen/pudding">a proof assistant</a> in Racket. This talk focused on the design and implementation of proof tactics.</p></li></ul> + +<p>After a short break, we heard about something completely different:</p> + +<ul> + <li><a href="http://justinpombrio.net/">Justin Pombrio</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/research/plt/dl/icfp2017/">ICFP 2017</a>) taught us to interpet the scoping rules of a &ldquo;core&rdquo; language as a preorder. Using the preorder, he then showed how to <em>infer</em> the scoping rules of any &ldquo;surface&rdquo; language based on its translation to the &ldquo;core&rdquo;.</li></ul> + +<p>Last summer and fall, Jeremy Siek hosted two REUs (<a href="https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5517&amp;from=fund">research experience for undergraduates</a>) at Indiana University. The two students gave the next talks:</p> + +<ul> + <li> + <p>Di Zhong (<a href="https://www.cs.indiana.edu/">IU</a>) talked about implementing interpreters in Racket, Python, and Haskell. As I understand, this was a hands-on experience through <a href="https://www.cis.upenn.edu/~bcpierce/tapl/">TAPL</a> and <a href="https://redex.racket-lang.org/">the Redex book</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="https://www.cs.indiana.edu/">IU</a>) demonstrated her implementation of <a href="http://theory.stanford.edu/~aiken/publications/papers/popl94.pdf">conditional types</a> for <a href="https://github.com/mvitousek/reticulated">Reticulated</a>.</p></li></ul> + +<p>Finally,</p> + +<ul> + <li><a href="https://nikivazou.github.io/">Niki Vazou</a> (<a href="https://www.cs.umd.edu/">UMD</a>) presented a theory of gradual refinement types. Any &ldquo;holes&rdquo; in the refinements introduce a search problem; type checking attempts to solve the problem by finding a predicate that unifies a function definition and its callers.</li></ul> + +<p>This meeting was a great opportunity to reflect on the recent past and share opinions on what&rsquo;s worth pursuing in the future. Many thanks to the participants, and to the NSF for the support!</p> + +<blockquote> + <p>If you want to know about the future, you need to ask the young people who will create it. Young people don&rsquo;t know what can&rsquo;t be done, and so they go ahead and do it. &mdash; <a href="https://www.youtube.com/watch?v=sM1bNR4DmhU">Ivan Sutherland</a></p></blockquote> + + Trees, 1973 + + urn:http-prl-ccs-neu-edu:-blog-2017-07-19-trees-1973 + 2017-07-19T21:48:56Z + 2017-07-19T21:48:56Z + + Ben Greenman + +<p>From the PRL archives:</p> + +<blockquote> + <p>I think that I shall never see a matrix lovely as a tree. &mdash; <a href="/img/gls-trees-poem-1979.pdf"><em>Trees</em></a>, by Guy L. Steele Jr., MIT, 1973</p></blockquote> +<!-- more--> + +<hr /> + +<p>You might recognize the opening line from Joyce Kilmer&rsquo;s 1914 poem <a href="https://en.wikipedia.org/wiki/Trees_(poem)"><em>Trees</em></a>, or from Radia Perlman&rsquo;s <a href="/img/p-sigcomm-1985.pdf"><em>Algorhyme</em></a> (published 1985).</p> + +<p>The poem is online in <a href="http://mercury.lcs.mit.edu/~jnc/humour/lisp.tree">at least one other place</a>, but the copy linked above (from <a href="https://archive.org/details/byte-magazine">BYTE magazine</a>) comes with a footnote on <em>How this poem came to be printed</em>.</p> + + Continuations + + urn:http-prl-ccs-neu-edu:-blog-2017-07-17-continuations + 2017-07-17T12:52:07Z + 2017-07-17T12:52:07Z + + Ben Greenman + +<p>From the PRL archives:</p> + +<blockquote> + <p>It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. &mdash; <a href="/img/nall-continuations-1983.pdf"><em>Continuations</em></a> by Alan Nall, Indiana University, 1983</p></blockquote> +<!-- more--> + +<hr /> + +<p>I first encountered this essay on continuations in a green folder in the PRL. It turns out, the author spent a semester at Indiana University working on the <a href="http://wiki.c2.com/?SameFringeProblem">same fringe problem</a> for a graduate-level programming languages course. According to <a href="https://www.cs.indiana.edu/~dfried/">the instructor</a>: &ldquo;What he said was true. He could not stop thinking about the problem the entire semester.&rdquo; This essay was a kind of final exam.</p> + + Quotes and Stories from "Turing 50" + + urn:http-prl-ccs-neu-edu:-blog-2017-06-24-quotes-and-stories-from-turing-50 + 2017-06-24T20:00:52Z + 2017-06-24T20:00:52Z + + Ben Greenman + +<p>The ACM recently hosted <a href="https://www.acm.org/turing-award-50">a celebration of 50 years of the A.M. Turing award</a>. These are some notes and thoughts from the event, including how Fred Brooks once rented a bus, Don Knuth&rsquo;s outrageous implementation of batch processing, and Judea Pearl&rsquo;s theory of homo sapiens.</p> +<!-- more--> + +<script>document.createElement('dialog');</script> + +<p><strong>Conventions / Disclaimers:</strong></p> + +<ul> + <li> + <p>The blockquotes below are paraphrased, may be incorrect, and may be incorrectly attributed. Make sure to watch the ACM&rsquo;s live stream before quoting anything here!!!</p></li> + <li> + <p>Section-breaks are labeled as &ldquo;panel&rdquo;, &ldquo;talk&rdquo;, &ldquo;question&rdquo;, etc.</p></li> + <li> + <p>This is intentionally &ldquo;bad writing&rdquo; in the Peter Lee sense (see below) &mdash; primarily &ldquo;what I saw&rdquo;, very little about &ldquo;what I thought and felt&rdquo;. A summary in my own words just wouldn&rsquo;t do justice to the panelists.</p></li> + <li> + <p>The &ldquo;Augmented Reality&rdquo; session was my favorite.</p></li></ul> + +<h3 id="opening-remarks">Opening Remarks</h3> + +<h4 id="alan-turing-is-with-us-today"><em>Alan Turing is with us today</em></h4> + +<p>At the start of the event, the <a href="http://users.ecs.soton.ac.uk/wh/">emcee</a> unveiled a bronze bust of Alan Turing. This statue was on display at center stage during the whole event.</p> + +<p>It&rsquo;s a good sculpture and it&rsquo;s good we remember Alan Turing, but I&rsquo;m sad that the ACM would encourage this kind of idol-worship. Let&rsquo;s not forget Turing&rsquo;s excellent teachers and colleagues!</p> + +<h3 id="talk-impact-of-turing-recipients-work">talk: Impact of Turing Recipients&rsquo; Work</h3> + +<h5 id="barbara-liskov">Barbara Liskov</h5> + +<blockquote> + <p><em>the first awards recognized achievements in the standard fields of theory, AI, and systems</em></p> + <p><em>hostile environment around the first awards, trepidation about future awards</em></p> + <p><em>with Unix, Ritchie and Thompson got the design right</em></p> + <p><em>Niklaus Wirth: &ldquo;If I understood how important Pcode was, I would have spent more time designing it&rdquo;</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>What is &ldquo;systems&rdquo; &mdash; does that even have a definition? And Unix is definitely NOT an example of a &ldquo;right design&rdquo;; rather it&rsquo;s a landmark of <a href="https://www.dreamsongs.com/WorseIsBetter.html">worse is better</a> design.</p> + +<h3 id="panel-advances-in-deep-neural-networks">panel: Advances in Deep Neural Networks</h3> + +<h4 id="stuart-russell">Stuart Russell</h4> + +<blockquote> + <p> <em>I work in all areas of AI except for deep learning</em></p></blockquote> + +<h4 id="judea-pearl">Judea Pearl</h4> + +<blockquote> + <p><em>I am a foreigner in this field &hellip; left because human beings are not good at handling information &hellip; people are very good with causal inference, not with statistical inference &hellip; deep learning is statistical</em></p> + <p><em>there is a very old existence proof, homo sapiens took over the planet &hellip; I believe because they had an internal model of their environment &hellip; a drawing of a lion with wings is evidence of this model, you have to have such a model before you can experiment with it and imagine &hellip; snakes have superb optics, result of a long evolution process &hellip; very specific but they cannot build eyeglasses &hellip; humans have an internal model, can build a market based on promises and build large communities based on promises</em></p> + <p><em>I see four levels &hellip; second level is predicting events, if I do X then what? &hellip; third level is counterfactual, if I did things differently then how would the outcome change &hellip; very hard to advance between levels, are we working to help machine learning &lsquo;level up&rsquo;?</em></p> + <p><em>data science is about the relation between data and reality &hellip; data alone is not data science</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>today we can&rsquo;t think without holding a piece of metal</em></p> + <p><em>machine learning is part of computer science rather than AI &hellip; AI is about how to make human &hellip; machine learning is about allocating resources &hellip; matrices are not all of human intelligence &hellip; neural nets are part of a wider toolbox &hellip; too much hype in NLP its just syntax</em></p> + <p><em>huge gap between syntax and semantics &hellip; chat bots are just syntax, don&rsquo;t learn &hellip; faking intelligence with neural nets, so well that you can build a company &hellip;</em></p> + <p><em>real metric is task completion</em></p> + <p><em>if I say &lsquo;a GLEEB walked across the airport&rsquo; then true intelligence can make a lot of educated guesses about a &lsquo;GLEEB&rsquo; without any other context</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>I disagree, ML is part of AI &hellip; understanding intelligence and making intelligent methods for solving AI problems</em></p> + <p><em>to quote Churchhill &lsquo;its not beginning of end, not end, not beginning of end, probably end of beginning&rsquo;</em></p> + <p><em>todays AI powered by hardware and data</em></p> + <p><em>AI cannot yet find our keys</em></p> + <p><em>quote: &lsquo;todays AI is making a perfect chess move while the world is on fire&rsquo; &hellip; ignores context</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>Turing &hellip; a program is a mathematical object &hellip; math community did not recognize this</em></p> + <p><em>lots of grad student descent &hellip; tuning to get performance &hellip; deep learning is neglecting the problem of exponential data &hellip; deep learning is just circuits, circuits lack expressive power &hellip; a human can process data from CERN but a neural net cannot, need to know physics</em></p> + <p><em>probabilistic programming, somewhat under the radar, maybe on the right track &hellip; 10-line program running/generating/updating a large network of possibilities &hellip; more composable and flexible</em></p></blockquote> + +<h5 id="ilya-sutskever">Ilya Sutskever</h5> + +<blockquote> + <p><em>why I like deep learning &hellip; philosophically satisfying &hellip; the hypothesis class is a circuit &hellip; powerful hypothesis class not too many parameters &hellip; can actually find circuits &hellip; &lsquo;violates all theory&rsquo; &hellip; really amazing &hellip; humans can see and hear pretty fast, even though our neurons are pretty slow, perhaps because we do a massively parallel process that doesn&rsquo;t take many steps &hellip; works well enough to be useful</em></p> + <p><em>models e.g. for vision are very hard to understand &hellip; fight fire with fire &hellip; incomprehensible solution to incomprehensible problem</em></p></blockquote> + +<h5 id="raquel-urtasun">Raquel Urtasun</h5> + +<blockquote> + <p><em>the breakthrough in neural nets is not algorithms &hellip; it is tricks, hardware, and grad students</em></p> + <p><em>with neural nets we forget about modeling, uncertainty, and prior knowledge &hellip; perception is a canonical example</em></p></blockquote> + +<h4 id="question-boundaries">question: boundaries</h4> + +<h5 id="judea-pearl">Judea Pearl:</h5> + +<blockquote> + <p><em>glad to see people in deep learning understand its limitations &hellip; is there a clearer definition of the boundaries? Are you worried about bridging the levels factual/inferential/counterfactural?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>the big problem is decision making under uncertainty</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>cognition is a hard problem</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>do you have a clear idea of the boundaries?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>neural nets use back-propagation &hellip; its non-modular, sad fact &hellip; performance and explainability is the tradeoff &hellip; then again people are non-modular</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>AlphaGo is not deep learning &hellip; basically an improved version of the machines Arthur Samuel made in the late 1950s &hellip; the interesting code is in C++ &hellip; rules of go, next moves, searching future states &hellip; depends on transitive closure</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>can AlphaGo take advice from a human?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>not currently, but that would be a new policy to add to the toolbox &hellip; just as neural nets are one tool within AlphaGo</em></p></blockquote> + +<h5 id="raquel-urtasun">Raquel Urtasun</h5> + +<blockquote> + <p><em>no reason to ask if deep learning is going to solve all problems</em></p></blockquote> + +<h4 id="question-education">question: education?</h4> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>indeed, what DO you teach in your neural networks classes?</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>&hellip; chain rule, Taylor expansion</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>teaching is communicating truths &hellip; what is true about neural nets? what are some things that will definitely not happen?</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>Peter Norvig and I have a problem with our AI book &hellip; chapter on vision, chapter on speech, will probably post just point to the neural nets chapter &hellip; we don&rsquo;t really understand! &hellip; really selling students short</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>in labs we talk about what we cannot do &hellip; we all have open problems</em></p> + <p><em>Stuart I hope you have a very good author for the chapters. There are so many open problems to communicate to students!</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>CS cirriculum needs more statistics, inferential thinking &hellip; revise the whole cirriculum bottom-up to weave this in</em></p></blockquote> + +<h4 id="question-could-a-neural-net-fix-my-phone-without-breaking-it">question: could a neural net fix my phone without breaking it?</h4> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>right! big problem that neural nets have no internal model to manipulate</em></p></blockquote> + +<h4 id="question-generalizability">question: generalizability?</h4> + +<h5 id="ilya-sutskever">Ilya Sutskever</h5> + +<blockquote> + <p><em>special-purpose vs. general purpose solution depends on the problem &hellip; most things we give special-purpose solutions &hellip; I guess if you wanted to automate a mathematician that would need to be general</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>always argue with your self &hellip; try to break what you&rsquo;ve built &hellip; there&rsquo;s a system that plays video games just using the pixels on screen as hints &hellip; it&rsquo;s very good at mazes; if a newborn baby learned to play maze games in 2 hours that would be amazing! &hellip; does the system scale? absolutely not</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>When Michael Jordan said &ldquo;people are non-modular&rdquo;, I think he means that people are able to break abstraction barriers when needed.</p> + +<h3 id="panel-restoring-personal-privacy-without-compromising-national-security">panel: Restoring Personal Privacy without Compromising National Security</h3> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>&hellip; wikileaks &hellip; russian hackers &hellip; social emergency &hellip;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>everything I say today is copyleft</em></p> + <p><em>its a misunderstanding to talk about a conflict between security and privacy &hellip; two aspects &hellip; problem goes back to feudalism &hellip; the right to build a castle was granted by the king &hellip; on one hand a castle improves national security &hellip; on the other hand a castle can be used to attack the king &hellip; technology is upsetting the basic notion of private vs. public security &hellip; governments cannot protect citizens and cannot protect themselves &hellip; extremely difficult to prove that a small process is secure</em></p> + <p><em>exceptional access makes it more complex</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>major concern are national security threats and ability of authorities to confound threats &hellip; analogy to printing press &hellip; proclimation of 1635 that only state messengers can carry letters &hellip; 1663 treatise by the national censor, no printing house can have a back door &hellip; the general topic is very old &hellip; title of this session isn&rsquo;t very good, the real dilemma is investigation vs privacy</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>code is law for better or worse, tech is not a tool like a watch &hellip; tech can monitor us and decide when it works &hellip; tech is government, not obedient tools &hellip; the mind is a warrant-proof space &hellip; 5th amendment rights should extend to wearables</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>cannot divorce the security/privacy issues from the current political context &hellip; the serious vulnerabilities are not in math &hellip; they are in users and implementors</em></p></blockquote> + +<h4 id="question-back-doors">question: back doors</h4> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>perhaps we should explain what a back door is</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>agency keeps a master key in escrow</em></p> + <p><em>non-lawyers can and should take a stand on basic issues</em></p> + <p><em>there are legitimate warrant-proof spaces &hellip; electronic extensions of the mind need to be recognized as warrant-proof spaces</em></p> + <p><em>the set of authorities with backdoor access should change as I travel between countries &hellip; but this will lead to a global race to the bottom</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>germany has a law against sex tourism (committed by German citizens visiting other countries) &hellip; neither government will be willing to lose backdoor access</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>technical reasons against backdoors &hellip; (1) &lsquo;weak crypto&rsquo; was implemented, nobody turned it off, is now breakable by anyone in 2015 &hellip; (2) Juniper used non-default crypto parameters, someone (inside?) changed the parameters &hellip; (3) attackers exploit back doors</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>quote &lsquo;you can put a man on the moon, surely you can put a man on the sun&rsquo;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>trouble is getting him back safely</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>I think back doors are okay, but not for personal devices &hellip; need public lab and transparent processes, need separation of powers &hellip; prosecutors are getting cases thrown out because courts do not accept their backdoors &hellip; there is a place for transparent back door tools</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>politicians are rarely technical people</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>tech is not a set of policy-neutral tools, need to address gap of understanding</em></p></blockquote> + +<h4 id="question-">question: ???</h4> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>we don&rsquo;t know how to build good crypto programs &hellip; opponents are debugging our programs with different goals &hellip; we&rsquo;re trying for-all-paths safety (universal) &hellip; they&rsquo;re trying exists-bad-path (existential)</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>cybersecurity market is a lemon market</em></p></blockquote> + +<h4 id="question-how-to-advise">question: how to advise</h4> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>question from audience &lsquo;I am an advisor to a company working with nuclear energy, they are terrified of being attacked, how should I advise them?&rsquo;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>a network like that is probably separated enough to be safe &hellip; the problem is being safe AND connected to the web</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>because the internet of things</em></p></blockquote> + +<h4 id="question-what-should-the-acm-do">question: what should the ACM do?</h4> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>maybe we need increased regulation, the ACM could help bring experts together</em></p></blockquote> + +<h4 id="question-what-is-true-security">question: what is true security</h4> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>it&rsquo;s all the same thing &hellip; gets labeled differently &hellip; just trying to control which bits can go where and who gets to read them</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>security is the absense of being violated</em></p></blockquote> + +<h5 id="paul-syverson-no-true--security-need-to-consider-context">Paul Syverson: <em>no true &gt; security, need to consider context</em></h5> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>problem of our community, have strict standards, may be unrealistic &hellip; maybe a lot more tolerance in practice than our model accepts</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>security and privacy are environmental problems</em></p></blockquote> + +<h4 id="question-can-we-stop-the-needle-in-haystack-search-for-vulnerabilities">question: can we stop the needle-in-haystack search for vulnerabilities?</h4> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>need to build in security from the start</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>need rule of law, transparency, separation of powers</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>stop delaying, instead of spending $$$ on fixing problems, we should invest in solving the fundamental issues</em></p></blockquote> + +<h3 id="panel-preserving-our-past-for-the-future">panel: Preserving our Past for the Future</h3> + +<p>Note: I was volunteering during this session; quotes are sparse</p> + +<h5 id="mahadev-satyanarayanan">Mahadev Satyanarayanan</h5> + +<blockquote> + <p><em>the running system is the total documentation &hellip; there are too many details for prose to capture</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>running old code has a danger of running old bugs</em></p></blockquote> + +<h5 id="">??:</h5> + +<blockquote> + <p><em>what not to save? &hellip; it&rsquo;s very hard to tell in advance</em></p></blockquote> + +<h5 id="mahadev-satyanarayanan">Mahadev Satyanarayanan:</h5> + +<blockquote> + <p><em>there is no absolute censor in a world with caching</em></p></blockquote> + +<h5 id="brewster-kahle">Brewster Kahle</h5> + +<blockquote> + <p><em>asking UNESCO to solve the problem is unrealistic &hellip; need to empower the fanatics, given them tools to preserve data</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>I totally agree with the &ldquo;empower the fanatics&rdquo; sentiment. Today, because of &ldquo;volunteer librarians&rdquo;, I think we&rsquo;re doing pretty well about preserving the past. Suppose I found an old PowerPoint file. I&rsquo;m sure I could find a way to read it with help from the internet &mdash; either by searching Google, pirating an old version of PowerPoint, or asking online forums. So personally I&rsquo;m not worried about losing data we have currently; I&rsquo;m more worried about the future, the internet becoming &ldquo;less chaotic&rdquo;.</p> + +<p>The panel raised a good question about how to preserve research and encourage reproducibility. A <code>.pdf</code> or <code>.tex</code> document is not enough; a virtual machine is okay. Really I think we need a stronger cultural emphasis on literate programming and a mature library like TeX to help authors store and share their work. <a href="https://thegamma.net/">The Gamma</a> seems on the right track.</p> + +<p>I was surprised that the panel did not discuss search, version control, and the ACM&rsquo;s open-access policy.</p> + +<h3 id="panel-moores-law-is-really-dead-whats-next">panel: Moore&rsquo;s Law is Really Dead: What&rsquo;s Next?</h3> + +<p>Note: I was volunteering during this session</p> + +<h5 id="butler-lampson">Butler Lampson</h5> + +<blockquote> + <p><em>there&rsquo;s plenty of room at the top &hellip; with Moore&rsquo;s Law we got improvements at the bottom of the software stack, everything above got to benefit and it was easy to integrate the changes &hellip; there&rsquo;s lots of opportunities to trim fat in the middle/top of the software stack &hellip; these improvements will be harder to integrate, but there&rsquo;s lots of opportunities</em></p></blockquote> + +<h5 id="margaret-martonosi">Margaret Martonosi</h5> + +<blockquote> + <p><em>By the way, don&rsquo;t believe the brochure that says I&rsquo;m at Google. My affiliation is Princeton, Google and I are just friends.</em></p></blockquote> + +<h5 id="butler-lampson">Butler Lampson</h5> + +<blockquote> + <p><em>important to distinguish approximate vs. precise software &hellip; precise software has a specification and the customer cares about that specification &hellip; approximate software doesn&rsquo;t have a hard spec, just needs to approximately work &hellip; the web is approximate, it doesn&rsquo;t work and it doesn&rsquo;t need to! &hellip; windows is precise, definitely has a spec and users definitely care</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>The recording of this panel should be good; it was very lively, very practical. And the first audience question (by <a href="https://people.eecs.berkeley.edu/~pattrsn/">David Patterson</a>) was &ldquo;an A+ question&rdquo;.</p> + +<p>The panel reminded me of a talk by <a href="http://users.ece.utexas.edu/~patt/">Yale Patt</a> about &ldquo;the end&rdquo; of the Von Neumann architecture. His opinion is future computers will be Von Neumann machines that rely on &ldquo;accelerators&rdquo; like a GPU &mdash; computer organization is not going to change, but will expand to have a bigger toolbox. So sure, Moore&rsquo;s Law is dead, but there are many opportunities to make computers faster at places other than the bottom of the software stack.</p> + +<h3 id="panel-challenges-in-ethics-and-computing">panel: Challenges in Ethics and Computing</h3> + +<p>Note: I was volunteering during this session</p> + +<h5 id="raj-reddy">Raj Reddy</h5> + +<blockquote> + <p><em>there are more slaves in the world currently than there were in the US during the civil war &hellip; here is one way technology could help, by giving everone a device to record their location &hellip; if someone&rsquo;s time and location is constant, they may be held against their will</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>do you believe every problem has a technological solution?</em></p></blockquote> + +<h5 id="noel-sharkey">Noel Sharkey</h5> + +<blockquote> + <p><em>yes the training set may be biased against people similar to me, but I want you to consider my case as an individual</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>a very nice Washington Post article</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>whether to encrypt the back hall</em></p></blockquote> + +<h5 id="raj-reddy">Raj Reddy</h5> + +<blockquote> + <p><em>we can sit here and wring our hands, but nothing will come of it unless it is written in the US constitution</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>I did not enjoy this panel. This is an ACM event, not a United Nations event. An ACM-sponsored panel about social and political problems should look for constructive ways that computer science can address these problems. Raj Reddy tried to give constructive solutions, but the panel seemed more interested in complaining about how hopeless things are.</p> + +<p>The comment by Noel Sharkey about &ldquo;consider me as an individual&rdquo; was something I hadn&rsquo;t thought about. Instead of worrying about biased datasets, let&rsquo;s use technology to collect data on an individual instead of abstracting a person by their race, age, or neighborhood.</p> + +<h3 id="talk-computer-science-as-a-major-body-of-accumulated-knowledge">talk: Computer Science as a Major Body of Accumulated Knowledge</h3> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p> <em>don&rsquo;t applaud me, just read my books</em></p> + <p><em>at the time, computer science was AI, numerical analysis, and programming languages</em></p> + <p><em>a colleague said &lsquo;I will believe that computer science is a science when it has 1000 deep theorems&rsquo; &hellip; I am not sure what a deep theorem is but I think its different from what&rsquo;s proven by deep learning</em></p> + <p><em>great privilege that we can invent the problems we work on &hellip; imagination &hellip; physicists can only guess the size of the sun</em></p> + <p><em>I&rsquo;ve always believed computer science and math are two parallel subjects &hellip; sometimes you hear people wondering if one subsumes the other</em></p> + <p><em>when I won the Turing Award, the prize money was about $1,000,000 less than it is today &hellip; I did get a nice Tiffany bowl that my wife and I use to serve strawberries &hellip; strawberries actually taste better &hellip;</em></p> + <p><em>very fortunate in this field &hellip; I&rsquo;m completely worthless as an economic advisor &hellip; it&rsquo;s a game I&rsquo;ve been able to take advantage of</em></p></blockquote> + +<h4 id="question-how-could-you-offer-to-pay-for-tex-bug-reports">question: how could you offer to pay for TeX bug reports?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>well there were many, many bugs &hellip; I stopped doubling at 32768 &hellip; brought people out of nowhere &hellip; next time I check the bug reports will be 2021 &hellip; someone is queueing the bugs reports &hellip; I believe strongly in batch rather than swap-in/swap-out &hellip; last time I checked reports was 2014 so 2021 will be next</em></p> + <p><em>TeX was a literate program, and it helped that I wrote &lsquo;The Errors of TeX&rsquo; about the first N bugs</em></p></blockquote> + +<h4 id="question-do-you-think-computers-will-become-good-composers-of-music-do-you-see-a-role-for-computer-assisted-proving">question: do you think computers will become good composers of music? do you see a role for computer-assisted proving?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>Yes in part, assisted is the key word &hellip; I have a program running now that I hope will help me prove a theorem</em></p></blockquote> + +<h4 id="question-favorite-algorithm">question: favorite algorithm?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>Tarjan&rsquo;s strong components &hellip; short deep useful</em></p></blockquote> + +<h4 id="question-thoughts-on-ai-computers-taking-over">question: thoughts on AI, computers taking over?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>I get scared when I see Stuart Russell making assumptions based on people acting rationally &hellip; then you look at election results</em></p></blockquote> + +<h4 id="question-if-you-could-start-over-and-do-things-differently-what-would-you-change">question: if you could start over and do things differently, what would you change?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>I would use decimal internally in TeX instead of binary</em></p></blockquote> + +<h4 id="question-how-to-record-history">question: how to record history?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>a video &lsquo;Lets not dumb down the history of CS&rsquo; &hellip; used to be history of algorithms &hellip; trouble with funding &hellip; the history is nothing that a non-CS person could not understand &hellip; the whole field of history changed from internal to external &hellip; historians need to be external to get published in journals &hellip; no CS department supports a historian &hellip; recently read a dissertation about the ALGOL 60 copmiler &hellip; very careful, describes data structures and organization &hellip; this kind of thing is what deserves to be called history</em></p></blockquote> + +<h4 id="question-teachers">question: teachers</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>hardest thing for me is choosing between two hypotheses (1) could teach this to anyone (2) only 2% of the world is geeks &hellip; suppose the second is true then you can&rsquo;t talk about how to teach if the teacher is not in the 2% &hellip;</em></p> + <p><em>the newest issue of CACM has a fun typo, &lsquo;deep earning&rsquo;</em></p></blockquote> + +<h3 id="panel-quantum-computing-far-away-around-the-corner-or-maybe-both-at-the-same-time">panel: Quantum Computing: Far Away? Around the Corner? Or Maybe Both at the Same Time?</h3> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>goal to have a 45&ndash;50 qbit machine &hellip; 1 error per 1000 operations &hellip; to test, run sample algorithm, chart output vs. a classical supercomputer &hellip; got to be a supercomputer to finish the computation in time</em></p></blockquote> + +<h5 id="andrew-yao">Andrew Yao</h5> + +<blockquote> + <p><em>I&rsquo;m a believer &hellip; one suggested benchmark is to factor 1000-digit numbers &hellip; impossible to attain &hellip; need to expore new possibilities, take physics attitute</em></p> + <p><em>CS did well attracting attention to quantum &hellip; science should be more open &hellip; share results between physics chemistry CS &hellip; don&rsquo;t just stick to your specialized conferences</em></p> + <p><em>CS departments reception to quantum is less than satisfactory &hellip; 15 years ago, maybe 4 or 5 universities &hellip; now, maybe 7 or 8 .. China doing much better in this regard</em></p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>not useful to make analogy to anything classical &hellip; universal fault tolerance? or computation in the presence of error &hellip; either would be excellent, still a long way off</em></p> + <p><em>IBM put quantum on the cloud &hellip; picked an instruction set that tries to abstract away &hellip; have been 19 published papers on the behavior of this quantum hardware</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>two paths &hellip; finding algorithms, besides Shor&rsquo;s algorithm &hellip; make quantum computer to realize the algorithms &hellip; finding algorithms is very difficult &hellip; information-processing point-of-view</em></p> + <p><em>error correction still small scale &hellip; can we use entanglement between probes to improve accuracy?</em></p></blockquote> + +<h5 id="umesh-vazirani">Umesh Vazirani</h5> + +<blockquote> + <p>_different goals &hellip; maybe you want perfect Qbits for a perfect Hilbert space &hellip; reality is a noisy space &hellip; short run, how to compute with noise &hellip; how to correct errors &hellip;</p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta:</h5> + +<blockquote> + <p>_those 2 paths are the same to me &hellip; we want larger devices with fidelity</p> + <p><em>lets build hardware see where goes &hellip; exciting prospect, computer scientists will explore what they can do with these erroneous qbits &hellip; that&rsquo;s why IBM has the instruction set open to the community</em></p></blockquote> + +<h4 id="question-why-isnt-adding-10-qbits-only-10x-harder">question: why isn&rsquo;t adding 10 qbits only 10x harder?</h4> + +<h5 id="john-martinis">John Martinis:</h5> + +<blockquote> + <p><em>building infrastructure to scale &hellip; not just grad student code &hellip; we&rsquo;re all good coders using standard industry practices for coding</em></p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>fidelity is hard to achieve</em></p></blockquote> + +<h4 id="question-both-ibm-and-google-use-superconducting-storage">question: both IBM and Google use superconducting storage?</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>superconducting scales &hellip; ion traps harder to scale, but we still watch, keep eye on data</em></p></blockquote> + +<h4 id="question-education">question: education</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>I like talking to engineering colleges &hellip; physics and engineering need to work together</em></p></blockquote> + +<h4 id="question-is-quantum-going-to-change-programing-languages">question: is quantum going to change programing languages?</h4> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>yes very different to handle errors &hellip; current challenge is building an abstraction over the superconducting hardware</em></p></blockquote> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>hoping to first expose hardware, then get a model, eventually a language</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>need to start with more algorithms</em></p></blockquote> + +<h4 id="question-what-would-feynman-do">question: what would Feynman do?</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>experiments!</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>yes he&rsquo;d tell us to keep playing, and play with us</em></p></blockquote> + +<h3 id="panel-augmented-reality-from-gaming-to-cognitive-aids-and-beyond">panel: Augmented Reality: From Gaming to Cognitive Aids and Beyond</h3> + +<p>Peter Lee starts off wearing a headset.</p> + +<h5 id="ivan-sutherland">Ivan Sutherland:</h5> + +<blockquote> + <p><em>I can tell you how VR started. Bell Helicopter company wanted to land at night &hellip; put an infrared camera on the landing site and a display in the cockpit &hellip; to test they used the roof of their building &hellip; one day an observer in a Bell office is watching, though the camera, two men playing catch on the roof &hellip; one player threw the ball at the camera and the observer ducked &hellip; he had identified his position with the camera &hellip; my observation was that you didn&rsquo;t need a camera, could substitute a computer &hellip; the rest is history</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>my goal is to augment people &hellip; <a href="https://en.wikipedia.org/wiki/The_Mother_of_All_Demos">Englebart</a> very inspiring &hellip; ok 2 stories &hellip; (1) a student of mine wanted to help picky eaters &hellip; computer vision for when they tried to hide peas under the plate &hellip; projected colors onto the peas, called them &lsquo;disco peas&rsquo;, kids were distracted enough to get over their aversion &hellip; children and parents got involved, new social interaction &hellip; (2) opera makeup for schoolchildren, virtually getting into character &hellip; teenage boys in the classes got to try makeup for the first time &hellip; singers found it useful for rehearsals</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>I feel socially awkward wearing this headset, but I have some of my slides here &hellip; making a wearable headset requires huge effort &hellip; research prototypes can be uncomfortable &hellip; a product needs to be perfect and its very hard to do perfect &hellip; one goal, give Lowe&rsquo;s VR to demo a virtual kitchen &hellip; Case Western anatomy class used virtual cadaver, great collective experience</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>two stories &hellip; (1) Henry Fuchs 1998, working with breast surgeon, try augmented reality to improve the precision of biopsy probe insertion &hellip; 2 years to a working prototype, hard to track surgeon&rsquo;s eyes, display where probe is, where ultrasound is, provide low latency &hellip; one day trying on live patient, worked 100% perfect probe right on the mark, jubilation &hellip; then the doctor had to tell the patient &lsquo;yes it is really cancer&rsquo; &hellip; (2) a challenge, augmented reality EMT training &hellip; real teams, virtual patient, virtual surround &hellip; track real tools, 8 eyes, 8 images, team needs to interact</em></p></blockquote> + +<h4 id="question-what-are-current-uses-of-augmented-reality">question: what are current uses of augmented reality?</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>the pilot of a jumbo jet typically has 1 hour flight experience before he flies for the first time, but extensive training in a flight simulator</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>the <strong>best</strong> AR</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>once I was in a flight simulator with the chief pilot &hellip; and he turned to me and asked &lsquo;have you ever experienced a slow roll in a 747?&rsquo; &hellip; a slow roll is a twisting motion, a very benign maneuver, constant one-G pressure the plane doesn&rsquo;t know its upside down &hellip; &lsquo;here we go&rsquo; and suddenly the world inverted &hellip; I remarked that it was certainly impressive, but didn&rsquo;t you treat the simulator as a real experience, and never attempted anything you would not do in reality? &hellip; &lsquo;that is true, but I am the chief pilot&rsquo;</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>construction, architecture</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>where&rsquo;s the &lsquo;augmented&rsquo;?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>whether augmented or virtual</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>yes we did my kitchen that way, made my wife sick when she tried it</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>surgical</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>still sounds virtual</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>displays on a car, superimposed directions on the tarmac &hellip; one of the users took a vacation and had to use the old GPS technology &hellip; found it very difficult to go back</em></p></blockquote> + +<h4 id="question-ar-tools-for-developers">question: AR tools for developers?</h4> + +<h5 id="blair-macintyre">Blair MacIntyre</h5> + +<blockquote> + <p><em>can developers write apps for the Microsoft <a href="https://www.microsoft.com/en-us/hololens">Hololens</a>?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>we belive in experience, anything we can do to foster experiences is good</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>faking things &hellip; subtle and important &hellip; I remember using a flight simulator, navigating the runway, and I turned my head to see if my wing was going to clip a plane &hellip; turned and there was nothing there &hellip; emotional shock to leave the simulation, I had been flying for 1 hour</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>pilot training is an early adopter because the cost of real planes is so high, impossible to train for emergency situations</em></p> + <p><em>the ultimate goal, you can sit in a virtual chair &hellip; and if the chair has handcuffs you cannot get up &hellip; a virtual bullet is lethal &hellip; probably impossible because bits don&rsquo;t weigh anything &hellip; you know Ben Franklin invented augmented reality, eyeglasses &hellip; the desire outweighs cost &hellip; I cannot see the audience here, maybe it would be good if I had a headset! but Peter took his off</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>because of my slides I couldn&rsquo;t see the audience, but then without the headset I couldn&rsquo;t see them either</em></p></blockquote> + +<h4 id="question-any-challenges-to-suggest-to-the-audience">question: any challenges to suggest to the audience?</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>if we had holographic transport, we wouldn&rsquo;t need planes!</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>maybe, but you need to give attendees a physical presence &hellip; smell, touch</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>what makes us willing to work together? I had a collaboration with three people &hellip; all in different locations .. communicated with a phone &hellip; worked perfectly, because we had worked in the same location first and knew one another so well &hellip; how to get to that point, where a simulation could be a useful tool &hellip; another good observation by Fred Brooks, given a domain X ask how good does the simulation need to be for X &hellip; Licklider told me, you&rsquo;d need damn good fiction to land someone on the moon, the simulation would need to provide every detail &hellip; for flight simulation the user&rsquo;s imagination can fill some gaps, a pilot can recognize an aircraft carrier from a rougher picture</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>at IBM I once hired buses to bring the Poughkeepsie secretaries to the main office &hellip; the secretaries at the two offices only knew one another from the phone &hellip; this one lunch did so much good &hellip; only $75 to rent a bus</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>how important is it to shake hands, to bump into a table?</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>for this conference, I think the live stream is getting a better experience because the cameras zoom in on us, the panelists &hellip; the audience in the back cannot see us, only a picture of us on the monitors</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p>_one excellent video game, starts in the dark, you hear a voice &hellip; turn around and there&rsquo;s a character sitting on a chair &hellip; if you rearrange your furniture he finds a new seat &hellip;</p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>games are a great example &hellip; Pokemon Go &hellip; Apple jusr released an app toolkit &hellip; need to get those in schools, in the hands of kids who can build with them</em></p></blockquote> + +<h4 id="question-ivan-about-your-ultimate-display-paper-what-has-since-surprised-or-frustrated-you">question: Ivan, about your &lsquo;ultimate display&rsquo; paper, what has since surprised or frustrated you?</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>I wasn&rsquo;t surprised because I never had any expectations &hellip; of course sticks are not real &hellip; no assumptions so no strong feelings</em></p></blockquote> + +<h4 id="question-people-already-distracted-by-cell-phones-how-to-manage-all-this-input">question: people already distracted by cell phones, how to manage all this input?</h4> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>good question, how much data you can present to people &hellip; and then the problem with google glass, your companions don&rsquo;t know what you are looking at &hellip; at least with snapchat glasses, you can trust the device is simpler</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>good writing defines reality, bad writing reports it &hellip; with the printing press, quickly went from 30,000 books to over 13,000,000 &hellip; novels evolved shortly after, a new form of expression</em></p></blockquote> + +<h4 id="question-peter-how-long-do-your-people-wear-the-hololens">question: Peter, how long do your people wear the hololens?</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>hard to say &hellip; but often longer than the battery lasts</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>how long does it last?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>depends what you&rsquo;re doing, 3 hours</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>that&rsquo;s encouraging, we had a 30-minute cutoff because participants had enough</em></p></blockquote> + +<h4 id="question-nausea">question: nausea</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>I get nauseous in our minecraft VR &hellip; but there&rsquo;s a pop-out feature where you keep playing, but the game world is in a TV set instead of around you &hellip; can pop back in when you&rsquo;re feeling better</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>we&rsquo;ve seen about 20% of the population gets nauseous</em></p></blockquote> + +<h5 id="audience-member">audience member</h5> + +<blockquote> + <p><em>Dana Boyd conducted an experiment, found the nausea was worse for wemon</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>oculus makes me feel sick, but the hololens has never given me trouble</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>have models to predict head motion, to keep the VR world steadier</em></p></blockquote> + +<h5 id="blair-macintyre">Blair MacIntyre</h5> + +<blockquote> + <p><em>I remember reading papers that measured framerate &hellip; would be interesting to revisit</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>framerate not important, its the latency that gets you &hellip; one colleague of mine, we call her &lsquo;the canary&rsquo; because she&rsquo;s so sensitive, in fact &hellip;</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>talking about nausea is part of the problem, people experience it more &hellip; every time I talk about it in public my co-workers tell me to stop!</em></p> + <p><em>another cool application, there&rsquo;s a hololens app to give blind people a tour of the Redmond office &hellip; you say a building and it takes you there</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>one challenge, the relative brightness of the real and virtual worlds</em></p></blockquote> + +<h4 id="question-any-last-remarks">question: any last remarks</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p>_I hoped from the beginning that AR would be a teaching tool &hellip; I learned that <code>F = MA</code> not from a book but from a large flywheel in the school&rsquo;s basement &hellip; very substantial inertia &hellip; the greatest value for AR would be to show people things in a way that makes the underlying meaning clear &hellip; what color should the hydrogen atoms in a benzene ring be? the color will be fiction, but the quality of learning will depend on that fiction &hellip; challenge for content makers &hellip; what is the haptic experience of feeling bits?</p></blockquote> + +<h3 id="small-group-session">Small Group Session</h3> + +<p>After the last panel, I got to attend a small group session with other students, Dick Karp, and Don Knuth. It doesn&rsquo;t feel right to summarize or quote from the session, but there&rsquo;s one thing I want to write about.</p> + +<p>During the group session, I said something that I now regret. There was a brief silence as the group changed subjects, and someone suggested that we do a round of introductions. I objected, <em>this will take so long</em>, but in fact the introductions were a very good idea.</p> + +<p>Normally, I don&rsquo;t like introductions because they focus on names, backgrounds, and credentials. I don&rsquo;t care about any of these when I&rsquo;m meeting someone! Rather, I prefer to just talk and by-the-way learn about the other person(s). There&rsquo;s an anaology to double-blind reviewing &mdash; the focus should be content and not credentials.</p> + +<p>These introductions were successful for two reasons. First, they gave everyone in the room a turn to speak, and this seemed to help people join the actual discussion sooner. That was strange to me. I always feel a little nervous the first time I speak up in front of a group, but if I really feel like speaking then I can always get over this little barrier. I guess it&rsquo;s not right to assume the nervousness is &ldquo;little&rdquo; for everyone. Second, the introductions format was &ldquo;say your name and a funny fact&rdquo;. This prompt by itself led to some nice conversation topics:</p> + +<ul> + <li>Could a computer program decide whether a statement was funny or not funny?</li> + <li>What kind of humor works in a classroom? In a textbook?</li> + <li>Would this kind of introduction be acceptable in another era or culture, for instance Victorian England?</li></ul> + +<p>&ldquo;Nice&rdquo; in the sense that everyone could contribute, which was a real challenge. Even the question &ldquo;does anyone have a favorite algorithm?&rdquo; didn&rsquo;t have much success fostering discussion.</p> + +<p>Related: a useful greeting at the event was &ldquo;what SIG are you?&rdquo;. The answer was a good hint about what level of abstraction you two could best communicate at.</p> +<!-- ### Misc.--> +<!-- I noticed that some of the young people who served on panels and also gave--> +<!-- long-and-not-very-insightful answers to questions were later on their laptops--> +<!-- as other panels discussed things. I noticed some of the older people who--> +<!-- served on panels falling asleep during other panels--> + + Building a Website with Scribble + + urn:http-prl-ccs-neu-edu:-blog-2017-05-23-building-a-website-with-scribble + 2017-05-23T01:53:13Z + 2017-05-23T01:53:13Z + + Ben Greenman + +<p>The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!</p> +<!-- more--> + +<h2 id="the-story-so-far">The Story so Far</h2> + +<p>Last Fall, I took a flight to Chicago (on my way to <a href="http://con.racket-lang.org/2016/">RacketCon 2016</a>). When I landed, there was a new message in my inbox:</p> + +<pre><code> Subject: Web Page + Date: 2016-09-15 + + You have been nominated webmaster by public acclamation. Congratulations!</code></pre> + +<p>Emboldened by the trust of my people, I promptly converted the PRL website from Racket-generating-HTML to the fine <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> preprocessor language (commit <a href="https://github.com/nuprl/website/commit/a0600d32fec4bd70c5530b2717aec32979d634f7"><code>a0600d</code></a>) This bold action polarized the community.</p> + +<blockquote> + <p>I can&rsquo;t read the source anymore! Is this really an improvement?</p></blockquote> + +<p>Fear not, citizens. The switch to <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> was the right choice, and you too can learn to read the source code.</p> + +<h2 id="how-to-read-scribblehtml-programs">How to Read <code>scribble/html</code> Programs</h2> + +<h3 id="basics">Basics</h3> + +<p>Scribble is a language for writing Racket documentation. The key innovation in Scribble is the <em>@-expression</em> (read: &ldquo;at expression&rdquo;). The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language combines @-expression syntax with functions that generate HTML.</p> + +<h4 id="-syntax">@-syntax</h4> + +<p><a href="http://www.greghendershott.com/2015/08/at-expressions.html">Greg Hendershott</a> and the <a href="http://docs.racket-lang.org/scribble/reader.html">Scribble Documentation</a> explain @-expressions properly. Here&rsquo;s a short tutorial (Part 1 of 2, &ldquo;the basics&rdquo;):</p> + +<ul> + <li>Scribble programs start in &ldquo;text mode&rdquo;. Every character you type goes straight to the document you are building.</li> + <li>The @-sign toggles to &ldquo;Racket mode&rdquo; for the next expression. In Racket mode, the characters you type will be evaluated as a Racket program to produce part of the document.</li></ul> + +<p><em>Examples:</em> Evaluating <code>"Hello Dave"</code> puts &ldquo;Hello Dave&rdquo; in your document. Evaluating <code>"Hello @Dave"</code> puts &ldquo;Hello ???&rdquo; in your document, where "???" is the value of the variable <code>Dave</code>. Finally if <code>Dave</code> is the name of a function, then <code>"Hello @(Dave)"</code> calls the <code>Dave</code> function with zero arguments and puts whatever it returns into your document.</p> + +<p>To make it easy to interleave text, function calls, and code, Scribble discriminates between 4 kinds of parentheses when they follow an @-sign (Part 2 of 2, &ldquo;the parens&rdquo;):</p> + +<ul> + <li><code>@(f A B)</code> is just like the function call <code>(f A B)</code> in Racket</li> + <li><code>@f[A B]</code> is the same as <code>@(f A B)</code>, but typically more useful because &hellip;</li> + <li><code>@f[A B]{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; to a list of words <code>w*</code>, then calls <code>f</code> just like <code>(apply f A B w*)</code></li> + <li><code>@f{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; and calls <code>f</code> with the results</li> + <li><code>@f|{....}|</code> is similar, but the <code>....</code> are in &ldquo;unescapable text mode&rdquo;</li></ul> + +<p>&ldquo;Unescapable text mode&rdquo; treats @-signs as text instead of toggling between modes.</p> + +<h4 id="generating-html">Generating HTML</h4> + +<p>The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language comes with functions that render HTML. These functions have the same name as the corresponding HTML tag.</p> + +<p>Example program:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this program prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>No surprises.</p> + +<p>One thing that <em>is</em> surprising is how <code>scribble/html</code> handles tag attributes. Every tag-rendering function accepts &ldquo;Racket mode&rdquo; arguments that specify an attribute name and attribute value.</p> + +<p>For example:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">[</span><span class="n">style:</span> <span class="s2">"color:red"</span><span class="p">]{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Hope the output looks familiar. The input syntax is strange, but that&rsquo;s what it is.</p> + +<p>Larger programs print larger webpages. Each page on the PRL website is HTML generated by one <code>scribble/html</code> program.</p> + +<h2 id="why-scribblehtml-is-an-improvement">Why <code>scribble/html</code> is an Improvement</h2> + +<p>Before <code>scribble/html</code>, the PRL website was implemented in <code>scribble/text</code>. A <code>scribble/text</code> program renders and prints text. There is no extra support for HTML.</p> + +<p>To compare, here&rsquo;s the start of the old homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/text</span> +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="s2">"templates.rkt"</span><span class="p">)</span> + +<span class="n">&lt;!DOCTYPE</span> <span class="n">html&gt;</span> +<span class="n">&lt;html</span> <span class="n">lang=</span><span class="s2">"en"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">header</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;body</span> <span class="n">id=</span><span class="s2">"pn-top"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">navbar</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;div</span> <span class="n">class=</span><span class="s2">"jumbotron"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And here is the start of the <code>scribble/html</code>&rsquo;d homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@require</span><span class="p">[</span><span class="s2">"templates.rkt"</span><span class="p">]</span> + +<span class="n">@doctype</span><span class="p">{</span><span class="n">html</span><span class="p">}</span> +<span class="n">@html</span><span class="p">[</span><span class="n">lang:</span> <span class="s2">"en"</span><span class="p">]{</span> + <span class="n">@header</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@body</span><span class="p">[</span><span class="n">id:</span> <span class="s2">"pn-top"</span><span class="p">]{</span> + <span class="n">@navbar</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@div</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"jumbotron"</span><span class="p">]{</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The pages look similar. The new one has more @-signs and parentheses, the old one has more <code>&lt;</code>-signs and quotes. If you were able to edit the old page, you should be able to edit the new page.</p> + +<p>The <strong>key improvement</strong> in the new page is that <strong>common mistakes are now compile-time errors</strong>.</p> + +<ul> + <li> + <p>Before, a typo like <code>&lt;hmtl&gt;</code> would generate an ugly webpage. After, a typo like <code>@hmtl</code> is a syntax error.</p></li> + <li> + <p>Before, a typo like <code>&lt;b&gt;....</code> with no closing tag would generate an ugly webpage. After, a typo like <code>@b{....</code> is a syntax error.</p></li></ul> + +<p>Both flavors of error message come with source-code line numbers. This is very very helpful.</p> + +<h3 id="small-improvements">Small Improvements</h3> + +<h4 id="1-more-functions">1. More Functions</h4> + +<p>Before, the <a href="http://prl.ccs.neu.edu/teaching.html">Teaching page</a> contained some interesting HTML for rendering vertical text (look for the word &ldquo;Semantics&rdquo; to see how this was used):</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"how-to-design-programs"</span><span class="p">&gt;</span>S<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>e<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>m<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>a<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>n<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>t<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>i<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>c<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>s<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>After, the same text is generated from a function call:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@span</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"how-to-design-programs"</span><span class="p">]{</span><span class="n">@vertical-text</span><span class="p">{</span><span class="n">Semantics</span><span class="p">}}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The <code>vertical-text</code> function is simple:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@require</span><span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._only-in))" style="color: inherit">only-in</a></span> <span class="n">racket/list</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span><span class="p">)]</span> + +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">vertical-text</span> <span class="o">.</span> <span class="n">str*</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-~3elist))" style="color: inherit">string-&gt;list</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._append*))" style="color: inherit">append*</a></span> <span class="n">str*</span><span class="p">))</span> <span class="p">(</span><span class="n">br</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="2-more-structure-less-boilerplate">2. More Structure, Less Boilerplate</h4> + +<p>Here&rsquo;s part of the old definition of &ldquo;Ben Greenman&rdquo; on the <a href="http://prl.ccs.neu.edu/people.html">People page</a>:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"row pn-person"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-row-eq-height"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-photo"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"img-wrapper"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">"img/ben_greenman.jpg"</span> <span class="na">title</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="na">alt</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="p">/&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-9"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-4 pn-contact"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"pn-name"</span><span class="p">&gt;</span>Ben Greenman<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + Advisor: Matthias Felleisen<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"mailto:types@"</span><span class="err">@"</span><span class="na">ccs</span><span class="err">.</span><span class="na">neu</span><span class="err">.</span><span class="na">edu</span><span class="err">"</span><span class="p">&gt;</span>types@"@"ccs.neu.edu<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"http://www.ccs.neu.edu/home/types"</span><span class="p">&gt;</span>www.ccs.neu.edu/home/types<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-muted col-md-offset-5"</span><span class="p">&gt;</span> + Joined 2014 + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-bio"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>I like constructions .... <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new definition uses a helper function with keyword arguments for each &ldquo;field&rdquo; of the person:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@person</span><span class="p">[</span><span class="kd">#:name</span> <span class="s2">"Ben Greenman"</span> + <span class="kd">#:title</span> <span class="s2">"Advisor: Matthias Felleisen"</span> + <span class="kd">#:e-mail</span> <span class="s2">"types@ccs.neu.edu"</span> + <span class="kd">#:website</span> <span class="s2">"http://ccs.neu.edu/home/types"</span> + <span class="kd">#:history</span> <span class="n">@list</span><span class="p">[</span><span class="s2">"Joined 2014"</span><span class="p">]</span> + <span class="kd">#:img</span> <span class="s2">"ben_greenman.jpg"</span><span class="p">]{</span> + <span class="n">I</span> <span class="n">like</span> <span class="n">constructions</span> <span class="n">....</span> +<span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="3-less-string-formatting">3. Less String-Formatting</h4> + +<p>Before, the code did a lot of string formatting (<a href="https://github.com/nuprl/website/commit/a0600d#diff-1921e33ce89be28dd277cf1c7880d1beL9">link</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/creatingunits.html#(form._((lib._racket/unit..rkt)._link))" style="color: inherit">link</a></span> <span class="n">url</span> <span class="n">body</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-append))" style="color: inherit">string-append</a></span> <span class="s2">"&lt;a href=</span><span class="se">\"</span><span class="s2">"</span> <span class="n">url</span> <span class="s2">"</span><span class="se">\"</span><span class="s2">&gt;"</span> <span class="n">body</span> <span class="s2">"&lt;/a&gt;"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new code has no need for such helper functions.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@a</span><span class="p">[</span><span class="n">href:</span> <span class="n">url</span> <span class="n">body</span><span class="p">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="bottom-line">Bottom Line</h4> + +<p>Scribble is a good language for making static HTML pages.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://docs.racket-lang.org/pollen/index.html">Pollen</a></li> + <li><a href="https://github.com/vishesh/racketscript">RacketScript</a></li> + <li>Other websites built using <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a>: (1) <a href="http://nanopass.org/">nanopass.github.io</a> (<a href="https://github.com/nanopass/nanopass.github.io">source code</a>), (2) <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> (<a href="https://github.com/nuprl/gtp">source code</a>).</li> + <li><a href="http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/">Notes from a Gradual Typing Across the Spectrum PI meeting</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Ben-Greenman.rss.xml b/blog/feeds/Author-Ben-Greenman.rss.xml new file mode 100644 index 00000000..bf360a7f --- /dev/null +++ b/blog/feeds/Author-Ben-Greenman.rss.xml @@ -0,0 +1,4381 @@ + + + + PRL Blog: Posts tagged 'Author: Ben Greenman' + PRL Blog: Posts tagged 'Author: Ben Greenman' + http://prl.ccs.neu.edu/blog/tags/Author-Ben-Greenman.html + Wed, 23 Dec 2020 18:21:55 UT + Wed, 23 Dec 2020 18:21:55 UT + 1800 + + Deep and Shallow Types + http://prl.ccs.neu.edu/blog/2020/12/23/deep-and-shallow-types/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-12-23-deep-and-shallow-types + Wed, 23 Dec 2020 18:21:55 UT + Ben Greenman + +<p>I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:</p> + +<ul> + <li><a href="http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020</a></li></ul> + +<p>To the PRL: thanks for a wonderful 6.5 years.</p> +<!-- more--> + +<h3 id="abstract">Abstract</h3> + +<blockquote> + <p>The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound&rsquo;&rsquo; mixed-typed languages from others that enforce type obligations locally rather than globally.</p> + <p>In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.</p> + <p>My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.</p> + <p>I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.</p></blockquote> + +<p>Next year, I&rsquo;ll be a <a href="https://cifellows2020.org">CI Fellow</a> at Brown.</p> + + Transient for Optional and Keyword Functions + http://prl.ccs.neu.edu/blog/2020/11/12/transient-for-optional-and-keyword-functions/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-11-12-transient-for-optional-and-keyword-functions + Thu, 12 Nov 2020 10:15:16 UT + Ben Greenman + +<p>A short adventure into the depths of optional and/or keyword functions in Racket.</p> +<!-- more--> + +<hr /> + +<p>Transient, or rather <em>the Transient semantics for a mixed-typed language</em>, is one way to let statically-typed code safely interact with untyped code. You can read all about it in <a href="http://hdl.handle.net/2022/23172">Michael Vitousek&rsquo;s 2019 dissertation</a> or <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my 2020 dissertation</a>, and you can see how it compares to other mixed-typed semantics <a href="http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">here</a>. The idea is to give up on <a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">behavioral type guarantees</a> and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called <em>shape checks</em>; for example:</p> + +<ul> + <li>if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;</li> + <li>if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and</li> + <li>if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.</li></ul> + +<p>Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type <strong>T</strong> into a shape check, and we have a function with type <strong>T</strong> that needs to check its inputs. The question is how to actually do the check in Racket v7.9.</p> + +<p>In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if <strong>T = (-&gt; Symbol Boolean)</strong> then we need to check the shape <strong>symbol?</strong> of the domain type <strong>Symbol</strong>:</p> + +<pre><code>;; source code +(: f (-&gt; Symbol Boolean)) +(define (f sym) + (eq? sym 'hola)) + +;; ===&gt; + +;; imaginary (but realistic) rewritten code +(define (f sym) + (assert sym symbol?) + (eq? sym 'hola))</code></pre> + +<p>A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type <strong>T</strong> accepts 1 to 3 inputs:</p> + +<pre><code>;; source code +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) +(define (g #:a a [b 'b] #:c [c #f]) + (if a b (if c 'left 'right))) + +;; ===&gt; + +;; imaginary, unrealistic rewritten code +(define (g #:a a [b 'b] #:c [c #f]) + (assert a boolean?) + (assert b symbol?) + (assert c void?) + (if a b (if c 'left 'right)))</code></pre> + +<p>Good &mdash; we basically know what we want. If the Racket core language had optional and keyword functions, then we&rsquo;d be done.</p> + +<p>But no, Racket expands these optional/keyword functions into primitive <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._lam))"><strong>lambda</strong></a> and <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._case-lam))"><strong>case-lambda</strong></a> forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.</p> + +<p>Let&rsquo;s keep digging.</p> + +<p>From now on, &ldquo;Shallow&rdquo; or &ldquo;Shallow TR&rdquo; refers to my implementation of Transient for Typed Racket (TR). We&rsquo;ll talk about Shallow instead of &ldquo;Transient&rdquo; in case future work reveals a better way to implement the Transient idea.</p> + +<h2 id="false-start-follow-the-type">False Start: Follow the Type</h2> + +<p>Beware &mdash; Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function <strong>g</strong> above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional <strong>#:c</strong> keyword lead to a shape-check failure because the variable <strong>c</strong> gets the default value <strong>#f</strong> instead of a void value. Concretely, the third assert from above fails:</p> + +<pre><code>(define (g #:a a [b 'b] #:c [c #f]) + .... + (assert c void?) ;; fails if c is the #f default value + ....)</code></pre> + +<p>The problem arises from subtyping. According to the annotations, the function <strong>g</strong> has an external type that is less precise than the internal type that validates the function body:</p> + +<pre><code>;; external type T +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) + +;; internal type T2, subtype of external (T2 &lt;: T), validates body +(: g (-&gt;* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))</code></pre> + +<p>Thanks to this external / internal distinction, the following easy rewrite idea, <em>Solution 0</em>, fails. Despite the failure, this first solution is a useful starting point for a success.</p> + +<h4 id="solution-0-step-1-mimic-the-typechecker">Solution 0, Step 1: Mimic the Typechecker</h4> + +<p>Shallow TR uses the same type checker as classic <em>Deep</em> TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.</p> + +<p>Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.</p> + +<p>Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (<a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-expr-unit.rkt#L274-L295">typecheck/tc-expr-unit.rkt</a>). The omitted code (<strong>&hellip;.</strong>) does actual type checking:</p> + +<pre><code>(define (tc-expr/check/internal form expected-type) + .... + (syntax-parse form + #:literal-sets (kernel-literals tc-expr-literals) + .... + [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^) + ....] + [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>Ok! Those two patterns say a lot about the expansion of optional and keyword functions:</p> + +<ol> + <li>Both forms expand to a <strong>let-values</strong> that binds one function <strong>fun</strong>.</li> + <li>TR uses the syntax classes <strong>kw-lambda^</strong> and <strong>opt-lambda^</strong> to tell these particular <strong>let-values</strong> apart from others.</li></ol> + +<p>Shallow TR can use exactly these patterns to recognize optional/keyword functions.</p> + +<h4 id="solution-0-step-2-parse-the-domain-type">Solution 0, Step 2: Parse the Domain Type</h4> + +<p>Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function&rsquo;s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:</p> + +<pre><code> [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/type-table.rkt#L80"><strong>type-of</strong></a> function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, <strong>fun</strong>, has a type. Great.</p> + +<p>Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:</p> + +<ul> + <li>The type of <strong>fun</strong> could be a straightforward <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L693"><strong>Fun type</strong></a>, but it could also be a: <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L701"><strong>DepFun type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L521"><strong>Poly type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L531"><strong>PolyDots type</strong></a>, or even a <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L900"><strong>Union type</strong></a>.</li> + <li>Each part of the domain type corresponds to one parameter of the <strong>fun</strong> expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in <strong>fun</strong>, or the mandatory keywords?</li></ul> + +<h4 id="solution-0-step-3-insert-a-shape-check">Solution 0, Step 3: Insert a Shape Check</h4> + +<p>Once we have the target <strong>fun</strong> expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse <strong>fun</strong> to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting <strong>fun</strong> goes something like this:</p> + +<pre><code>(syntax-parse fun + [(#%plain-lambda formals . body) + #:with (shape-check ...) + (make-shape-checks #'formals (type-of fun)) + #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])</code></pre> + +<p>The rewritten function executes shape checks immediately, and then proceeds with the <strong>body</strong> after validating each actual parameter.</p> + +<h2 id="on-the-trail-optkey-expansion">On the Trail: optkey Expansion</h2> + +<p>Our <em>Solution 0</em> fails because the type of the <strong>fun</strong> expression that it gets from the type-checked code is an external type. In terms of the <strong>g</strong> function from above, <em>Solution 0</em> uses the type <strong>Void</strong> instead of the internal type <strong>(U Void #f)</strong> to check the <strong>c</strong> parameter. To get internal types, we need to look closer at <strong>fun</strong> and the rest of the optional/keyword expansion.</p> + +<p>Let&rsquo;s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.</p> + +<p>If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function <strong>X</strong> create a module <strong>test.rkt</strong> like this:</p> + +<pre><code>#lang racket/base + +(define _ignore + (let () + X + (void)))</code></pre> + +<p>Invoke the expander with <code>raco expand test.rkt &gt; test.rkt.txt</code> and explore the generated <strong>.txt</strong> file.</p> + +<h3 id="example-1-mandatory-keyword">Example 1: mandatory keyword</h3> + +<p>The source is a function with one mandatory positional argument and one optional positional argument.</p> + +<pre><code>(lambda (x [y 0]) + (+ x y))</code></pre> + +<p>Expansion generates a <strong>case-lambda</strong> that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function <strong>F</strong> that expects two arguments, resolves defaults in a different way, and executes the function body.</p> + +<pre><code>(let-values (((F) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F x '0)) + ((x y1) (#%app F x y1))))</code></pre> + +<p>Note: the expression <strong>(if &rsquo;#f &rsquo;0 y1)</strong> in the generated <strong>F</strong> function is equal to <strong>y1</strong> alone. In general, the <strong>if</strong> is for default expressions. (<a href="https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/">Unlike Python</a>, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a <strong>#f</strong> test. A general-purpose optimizer can remove this test before the code runs.</p> + +<h3 id="example-2">Example 2:</h3> + +<p>The source is a function with one mandatory positional argument and one mandatory keyword argument:</p> + +<pre><code>(lambda (x #:y y) + (+ x y))</code></pre> + +<p>Expansion generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> expects a list of keywords, a list of arguments, and a final argument. The purpose of <strong>F1</strong> is to organize a call to <strong>F0</strong>.</li> + <li><strong>lifted/2</strong> is the constructor for a generated struct type. Other functions help the struct call <strong>F1</strong>. Nevermind the details; I don&rsquo;t fully understand them either.</li></ul> + +<p>The important piece for Shallow TR is the <strong>F0</strong> function because the goal of rewriting is to protect the original function body against untyped inputs.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) y1)) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y1) (#%app car given-args))) + (#%app F0 y1 x3))))) + (#%app + lifted/2 + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l2571) given-kws)) + (if (#%app pair? l2571) + (if (#%app eq? (#%app car l2571) '#:y) + (#%app null? (#%app cdr l2571)) + '#f) + '#f)) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + '(#:y) + '(#:y))))</code></pre> + +<h3 id="example-3">Example 3:</h3> + +<p>The source is a function with one mandatory positional argument and one optional keyword argument:</p> + +<pre><code>(lambda (x #:y [y 0]) + (+ x y))</code></pre> + +<p>Expansion again generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments, resolves the optional default, and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> calls <strong>F0</strong></li> + <li>At the bottom, there are two <strong>case-lambda</strong> functions that call <strong>F1</strong></li></ul> + +<p>Again, the <strong>F0</strong> function is the focal point for Shallow TR rewriting.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y2) (#%app pair? given-kws))) + (let-values (((y1) + (if y2 (#%app car given-args) '0))) + (#%app F0 y1 x3)))))) + (#%app + make-optional-keyword-procedure + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l1571) given-kws)) + (let-values (((l1571) + (if (#%app null? l1571) + l1571 + (if (#%app eq? (#%app car l1571) '#:y) + (#%app cdr l1571) + l1571)))) + (#%app null? l1571))) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + null + '(#:y) + (case-lambda + ((x) (#%app F1 null null x))))))</code></pre> + +<h2 id="solution-the-shallow-tr-rewrite-strategy">Solution: The Shallow TR Rewrite Strategy</h2> + +<p>All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a <strong>let-values</strong> form:</p> + +<pre><code>(let-values (((f) fun)) . body)</code></pre> + +<p>Furthermore, the generated <strong>fun</strong> is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the <strong>fun</strong> from <em>Example 3</em> again; it has formal parameters for the keyword arg. and the mandatory arg., and one <strong>let-values</strong> to resolve each parameter:</p> + +<pre><code> (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y)))))</code></pre> + +<p>Another experiment with <strong>type-of</strong> shows that the right-hand side of each <strong>let-values</strong> has an internal type annotation. Excellent! Both <strong>(type-of x3)</strong> and <strong>(type-of (if &rsquo;#f &rsquo;0 y1))</strong> are the right types for shape checks. Shallow TR can:</p> + +<ul> + <li>inspect the <strong>let-values</strong> one-by-one;</li> + <li>convert the type of each right-hand expression to a shape predicate; and</li> + <li>rewrite each right-hand <strong>expr</strong> into <strong>(assert expr shape?)</strong>.</li></ul> + +<p>This should work! In fact, we can do slightly better:</p> + +<ul> + <li>when the right-hand expression is a conditional <strong>(if test default-expr supplied-arg)</strong></li> + <li>then Shallow only needs to check the supplied arg: <strong>(if test default-expr (assert supplied-arg shape?))</strong></li></ul> + +<p>Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.</p> + +<h2 id="a-problem-with-methods-and-a-bugfix">A Problem with Methods and a Bugfix</h2> + +<p>Currently, Shallow TR rewrites optional and keyword functions using the <strong>let-values</strong> plan described above. Each formal parameter has one <strong>let-values</strong> binding, and the type on each bound expression defines the shape check.</p> + +<p>Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we <a href="https://github.com/racket/racket/pull/3182">fixed the class expander</a>.</p> + +<p>First, here is a class with one method that runs correctly. The method <strong>f</strong> accepts an optional positional argument <strong>x</strong>; the default value of <strong>x</strong> is the current value of the field <strong>my-num</strong> (fields are mutable):</p> + +<pre><code>(define c0% + (class object% + (super-new) + (field (my-num 2)) + (define/public (f [x my-num]) + (+ x x))))</code></pre> + +<p>Second, here is a similar method that fails. This time, the default is an immediate value <strong>2</strong>:</p> + +<pre><code>(define c1% + (class object% + (super-new) + (define/public (f [x 2]) + (+ x x))))</code></pre> + +<p>Running a call <strong>(send o1 f)</strong> used to raise a shape-check failure about a strange value:</p> + +<blockquote> + <p>shape error: Expected a real number, got <code>#&lt;unsafe-undefined&gt;</code></p></blockquote> + +<p>What is going on?</p> + +<p>It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:</p> + +<pre><code>(lambda (x [y z]) + (+ x y))</code></pre> + +<p>Expansion generates a function <strong>F0</strong> that checks for the undefined value, and an outer <strong>case-lambda</strong> that supplies undefined when the default is needed:</p> + +<pre><code>(let-values (((F0) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) + (if (#%app eq? y1 unsafe-undefined) + z + y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F0 x unsafe-undefined)) + ((x y1) (#%app F0 x y1))))</code></pre> + +<p>That&rsquo;s the normal way that <strong>unsafe-undefined</strong> shows up: the <a href="https://github.com/racket/racket/blob/c0ff11e27bd28e070c20b7a9b0f7365f8f2b665a/racket/collects/racket/private/kw.rkt">expander for optional/keyword functions</a> looks for default expressions vs. default values and uses the undefined value for expressions.</p> + +<p>Three other facts conspired to make the problem with optional methods:</p> + +<ol> + <li>Typed Racket also looks for default expressions vs. default values (search for <strong>immediate-default</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/base-env/annotate-classes.rkt">here</a>). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the <strong>unsafe-undefined</strong> value (search for <strong>-Unsafe-Undefined</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/kw-types.rkt">here (kw)</a> and <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-lambda-unit.rkt">here (opt)</a>).</li> + <li>The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.</li> + <li>Shallow TR pushes default expression checks <strong>(if test default-expr supplied-arg)</strong> to the <strong>supplied-arg</strong> instead of wrapping the whole <strong>if</strong> form.</li></ol> + +<p>In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound&mdash;but harmless because the false assumption was guarded by an <strong>if</strong> test for <strong>unsafe-undefined</strong>. Running Shallow TR revealed the unsoundness with its eager shape check.</p> + +<p>Again, the resolution was to fix the class expander (<a href="https://github.com/racket/racket/pull/3182">racket/racket #3182</a>). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.</p> + +<h2 id="lessons">Lessons</h2> + +<ol> + <li>Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.</li> + <li>Digging into the expansion is sometimes necessary. There are at least three places that do so&mdash;the class expander, TR, and Shallow TR&mdash;and unfortunately they all need to cooperate.</li> + <li>The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my dissertation</a> lists them all.</li></ol> + + Transient Answers Old Questions + http://prl.ccs.neu.edu/blog/2020/10/15/transient-answers-old-questions/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-10-15-transient-answers-old-questions + Thu, 15 Oct 2020 13:32:12 UT + Ben Greenman + +<p>Several old questions from the Typed Racket mailing list have new and simple answers under a &ldquo;transient&rdquo; Typed Racket.</p> +<!-- more--> + +<hr /> + +<p>For the past few months, I&rsquo;ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the <a href="https://github.com/racket/typed-racket/pull/952">RFC</a> and <a href="https://github.com/racket/typed-racket/pull/948">pull request</a>.</p> + +<p>The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that&rsquo;s about it. By contrast, types are much stronger in classic Typed Racket.</p> + +<p>Shallow Racket&rsquo;s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the <a href="https://groups.google.com/g/racket-users">Racket mailing list</a>. There were a surprising number of questions that went like this:</p> + +<blockquote> + <p><strong>Q.</strong> Hey, I ran a program expecting <em>X</em> to happen, but <em>Y</em> happened instead. Is this a bug?</p> + <p><strong>A.</strong> No, Typed Racket has to do <em>Y</em> because of its strong types.</p></blockquote> + +<p>&hellip; but changing to shallow types gives the <em>X</em> behavior! Here are their stories.</p> + +<p>Going forward, <strong>Deep</strong> refers to normal Typed Racket and <strong>Shallow</strong> refers to Shallow Typed Racket.</p> + +<hr /> + +<h2 id="higher-order-value-as-any">Higher-Order Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ">groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ</a></p> + +<h4 id="on-20180416-mailoo-wrote">On 2018&ndash;04&ndash;16, <em>mailoo</em> wrote:</h4> + +<blockquote> + <p> I play a little with the &ldquo;Any&rdquo; type (due to &lsquo;dynamic-require&rsquo; which return Any), and I&rsquo;m not able to cast them back in a function.</p> + <p> I (over) simplify my question with this little program :</p></blockquote> + +<pre><code>(: p Any) +(define (p i) (displayln i)) + +; Here I want to get back my function +(define proc (cast p (-&gt; Integer Void))) +(proc 2) </code></pre> + +<blockquote> + <p> but I get this error when I try to execute the function :</p></blockquote> + +<pre><code>; contract violation +; Attempted to use a higher-order value passed as `Any` in untyped code: #&lt;procedure:p&gt; </code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> raises an error because it must enforce the <code>Any</code> type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> prints 2 and returns void. No error. Same goes for dynamic-require.</p> + +<hr /> + +<h2 id="parametric-contract-affects-untyped-code">Parametric Contract Affects Untyped Code</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ">groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ</a></p> + +<h4 id="on-20191215-john-clements-wrote">On 2019&ndash;12&ndash;15, John Clements wrote:</h4> + +<blockquote> + <p> It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:</p></blockquote> + +<pre><code> #lang typed/racket + + (require/typed racket/list + [index-of (All (T) ((Listof T) T -&gt; (U False Natural)))]) + + (index-of '(n s e w) 'n) ;; returns... #f? </code></pre> + +<blockquote> + <p> In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).</p> + <p> I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces parametricity for <code>All</code> types, and this throws off the equality function that index-of uses internally.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> returns 0.</p> + +<p>ps John, thanks very much for working on <a href="https://adventofcode.com">Advent of Code</a> and mailing the list!</p> + +<hr /> + +<h2 id="unable-to-protect-opaque-value-as-any">Unable to Protect Opaque Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ">groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ</a></p> + +<h4 id="on-20191211-marc-kaufmann-wrote">On 2019&ndash;12&ndash;11, Marc Kaufmann wrote:</h4> + +<blockquote> + <p>I have one file called <code>type-test.rkt</code> with the following</p></blockquote> + +<pre><code>#lang typed/racket + +(require (only-in typed/web-server/http response/xexpr response)) + +(provide f2) + +(: f2 (-&gt; (U response Any))) +(define (f2) + (define x '(body (h1 "Try it"))) + (: resp response) + (define resp (response/xexpr x)) + resp)</code></pre> + +<blockquote> + <p>Then I have another <em>untyped</em> file for a servlet:</p></blockquote> + +<pre><code>#lang racket + +(require "type-test.rkt" + web-server/servlet + web-server/servlet-env) + +(define (start req) + (f2)) + +(serve/servlet start + #:servlet-regexp #rx"" + #:launch-browser? #false + #:port 8080)</code></pre> + +<blockquote> + <p>Notice that I am telling [f2] that <code>resp</code> is of type <code>response</code>. Yet, when I run the server with <code>start</code> [&hellip;.] I get the following result:</p> + <p>(f2): Error, see below.</p> + <p>The error is:</p></blockquote> + +<pre><code>f2: broke its own contract + any-wrap/c: Unable to protect opaque value passed as `Any` + value: #&lt;response&gt; + in: the range of + (-&gt; Any)</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the <code>Any</code> type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.</p> + +<hr /> + +<h2 id="type-inference-installs-a-precise-type">Type Inference Installs a Precise Type</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ">groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ</a></p> + +<h4 id="on-20200214-john-clements-wrote">On 2020&ndash;02&ndash;14, John Clements wrote:</h4> + +<blockquote> + <p>I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.</p> + <p>Here’s a program:</p></blockquote> + +<pre><code>#lang typed/racket + +(define-type Store (Mutable-HashTable Integer Value)) +(define-type Value (U Real Boolean String)) + +(define top-store + (cast + (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f))) + Store)) + +(hash-set! top-store 5 1234)</code></pre> + +<blockquote> + <p>It fails with this error:</p></blockquote> + +<pre><code>contract violation +expected: (or/c (and/c byte? positive?) #t #f) +given: 1234 +in: the values of +the 3rd conjunct of +(and/c hash? + hash-mutable? + (hash/c exact-integer? + (or/c (and/c byte? positive?) #t #f) + #:immutable #f))</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p>First off, <strong>Deep</strong> runs fine after swapping <code>cast</code> for <code>ann</code>.</p> + +<p>Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.</p> + +<p>The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.</p> + +<hr /> + +<h2 id="same-arity-functions-in-a-case-lambda">Same-Arity Functions in a Case Lambda</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ">groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ</a></p> + +<h4 id="on-20190705-ryan-kramer-wrote">On 2019&ndash;07&ndash;05, Ryan Kramer wrote:</h4> + +<blockquote> + <p>In the code below, can <code>maybe-car</code> have the given type [&hellip;.]?</p></blockquote> + +<pre><code>#lang typed/racket + +(module untyped racket + (provide maybe-car) + (define (maybe-car x) + (cond + [(pair? x) (car x)] + [else x]))) + +(require/typed + 'untyped + [maybe-car (All (a b) (case-&gt; + (-&gt; (Pairof a b) a) + (-&gt; a a)))])</code></pre> + +<blockquote> + <p>[Current error:]</p></blockquote> + +<pre><code>Type Checker: + Type (All (a b) (case-&gt; (-&gt; (Pairof a b) a) (-&gt; a a))) + could not be converted to a contract: + function type has two cases of arity 1</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the type with a Racket <code>or/c</code> contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don&rsquo;t know which to apply to an incoming function.</p> + +<p>Note, the &ldquo;Type Checker&rdquo; error message is much better than what or/c would give on its own.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> simply checks that maybe-car accepts both arities inside the case-&gt; type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.</p> + +<hr /> + +<h3 id="immutable-type-affects-untyped-code">Immutable Type Affects Untyped Code</h3> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ">groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ</a></p> + +<h4 id="on-20200217-bertrand-augereau-wrote">On 2020&ndash;02&ndash;17, Bertrand Augereau wrote:</h4> + +<blockquote> + <p>Hello everybody, I&rsquo;m trying to gradually type my script to make it a proper app (yes I&rsquo;m a static-ish guy) and I have an issue (Racket 7.6 CS).</p></blockquote> + +<pre><code>; racket_mod.rkt: +#lang racket + +(provide (struct-out s)) +(provide list-of-s) +(provide set-list-of-s!) + +(struct s (a)) +(define list-of-s '()) +(define (set-list-of-s! los) + (set! list-of-s los))</code></pre> + +<pre><code>; racket_mod_typed.rkt: +#lang typed/racket + +(provide (struct-out s2)) +(provide list-of-s2) +(provide set-list-of-s2!) + +(struct s2 ([a : Natural])) +(define list-of-s2 '()) +(define (set-list-of-s2! [los : (Listof s2)]) + (set! list-of-s2 los))</code></pre> + +<pre><code>; racket_main.rkt: +#lang racket + +(require "racket_mod.rkt") +(require "racket_mod_typed.rkt") + +(define los (list (s 1) (s 2))) +(set-list-of-s! los) +(displayln list-of-s) + +(define los2 (list (s2 1) (s2 2))) +(set-list-of-s2! los2) +(displayln list-of-s2)</code></pre> + +<blockquote> + <p>list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces the type of <code>list-of-s2</code> with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.</p> + +<p>Here&rsquo;s a step-by-step breakdown:</p> + +<ol> + <li>the typed module creates an empty list-of-s2</li> + <li>the main module imports the list and receives a new copy</li> + <li>the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable</li> + <li>the main module reads from its copy &mdash; and it&rsquo;s still empty</li></ol> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> lets the original list travel to untyped code. There are no contracts in the way.</p> + +<h2 id="discussion">Discussion</h2> + +<p>Wow! It&rsquo;s great to see that Shallow Racket works &ldquo;as expected&rdquo; on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.</p> + +<p>If you have a similar experience with a deep-types error, let me know.</p> + +<p>Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error &mdash; and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.</p> + +<p>Shallow Typed Racket is coming soon. Follow the <a href="https://github.com/racket/typed-racket/pull/948">pull request</a> or watch the Racket release notes for news.</p> + +<h3 id="links">Links</h3> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">Larger example</a> where Shallow misses an error that Deep catches</li> + <li>Michael M. Vitousek <a href="http://hdl.handle.net/2022/23172">invented</a> the Transient semantics and implemented it in <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>.</li> + <li>My <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-thesis-2020">upcoming dissertation</a> has lots more to say about Shallow Typed Racket.</li></ul> + +<p><em>Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.</em></p> + + The Typed Racket Optimizer vs. Transient + http://prl.ccs.neu.edu/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + Wed, 15 Jan 2020 12:16:35 UT + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> + + PRL Offsite 2019 Retrospective + http://prl.ccs.neu.edu/blog/2019/12/12/prl-offsite-2019-retrospective/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-12-12-prl-offsite-2019-retrospective + Thu, 12 Dec 2019 12:51:53 UT + Ben Greenman, Olek Gierczak + +<p>On November 11th 2019, the PRL had a private offsite meeting at the <a href="https://mtwyouth.org">More Than Words bookstore</a> in downtown Boston. For future offsite organizers, this post records what happened and how.</p> +<!-- more--> + +<h2 id="early-planning-goals">Early Planning, Goals</h2> + +<p>Every Fall, the PRL holds a kickoff meeting to assign <a href="http://prl.ccs.neu.edu/contact">roles</a> for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.</p> + +<blockquote> + <p>Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the <a href="http://www.warrencenter.com">Warren Conference Center</a> is <a href="https://www.northeastern.edu/events/northeastern-owned-off-campus-venues">Northeastern University&rsquo;s only franchised</a> off-campus venue.</p></blockquote> + +<p>The <strong>motivation</strong> for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students &mdash; both to get to know the group, and to become known. They&rsquo;re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">Spring 2017 edition</a> of HOPL.</p> + +<p>Our primary <strong>goal</strong> this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.</p> + +<p>In light of the main goal, we prefer to call the event an &ldquo;offsite&rdquo; (or &ldquo;offsite meeting&rdquo;) rather than a &ldquo;retreat&rdquo; because the target was an informative day rather than a relaxing one.</p> + +<h2 id="booking-and-logistics">Booking and Logistics</h2> + +<p>Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that&rsquo;s all we had.</p> + +<h3 id="date--nov-11">Date = Nov 11</h3> + +<p>In the beginning, we chose Veterans&rsquo; Day (2019&ndash;11&ndash;11) and Northeastern <a href="https://registrar.northeastern.edu/app/uploads/2019-2020-University-Wide-Calendar-List.pdf">reading day</a> (2019&ndash;12&ndash;05) as possible dates. We ended up with Veterans&rsquo; Day.</p> + +<p>A small number of lab members were opposed to Veterans&rsquo; Day. They gave two reasons: the Fall semester is especially busy, and Veterans&rsquo; Day is a federal holiday.</p> + +<h3 id="venue--mtw">Venue = MTW</h3> + +<p>The first venue we tried was the <a href="http://www.warrencenter.com">Warren Conference Center</a> in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the <a href="https://www.khoury.northeastern.edu/people/chelsea-smith/">Khoury Events Team</a> to reach out on our behalf. They returned with an email address and <a href="/blog/static/warren-center-meetings-and-retreats.pdf">event menu</a> (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.</p> +<!-- Donald Pepple (Northeastern) made contact with Christine Barisano (Framingham)--> + +<p>Second, we contacted the <a href="https://alumni.northeastern.edu/about/alumni-center/">Northeastern University Alumni Center</a>. They were not open on Veterans&rsquo; Day 2019.</p> + +<p>Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: <a href="https://mtwyouth.org">More Than Words</a>.</p> + +<p>More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!</p> + +<blockquote> + <p>More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you&rsquo;ll see a mix of &ldquo;employees&rdquo; and &ldquo;volunteers&rdquo; helping out.</p></blockquote> + +<p>Booking was complicated, though, by the fact that Northeastern requires <a href="http://catalog.northeastern.edu/graduate/health-sciences/academic-policies-procedures/liability-insurance/">liability insurance</a> for all off-campus events. If <strong>you</strong> are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.</p> + +<p>Here are more facts about MTW:</p> + +<ul> + <li>the theater space seats 30 with lots of extra space</li> + <li>the two long tables in the reading room can seat about 20</li> + <li>the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor</li> + <li>there&rsquo;s no whiteboard, but MTW can supply an easel stand</li> + <li>much of the furniture in store is antique and for sale, so be careful using it &mdash; both to avoid damaging it, and because antiques can be oddly-shaped</li> + <li>the bookstore was closed on Veterans&rsquo; Day, but we were able to have our event and buy books</li> + <li>MTW may have fridge space to temporarily hold leftovers</li> + <li>closest T station: <a href="https://www.mbta.com/stops/place-tumnl">Tufts Medical Center</a></li></ul> + +<h3 id="keynote--none">Keynote = None</h3> + +<p>The original, ambitious plan was to invite two keynote speakers &mdash; one working in industry and one from academia &mdash; to enrich the offsite with new knowledge. And because this was the PRL&rsquo;s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)</p> + +<p>We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.</p> + +<p>Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).</p> +<!-- to future planners: we invited Kathi Fisler; she had grant-writing plans for that day and would be interested in coming to a future offsite--> + +<h3 id="food-coffee-tea">Food, Coffee, Tea</h3> + +<p><a href="https://flourbakery.com/">Flour Bakery + Cafe</a> provided lunch. For most days, you can order from Flour using an <a href="https://flourbakery.com/menu/catering/catering-information/">online form</a>. For holidays, you may need to send an email &mdash; that&rsquo;s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan &amp; gluten-free <em>everything spiced</em> salads.</p> + +<p><a href="https://www.trycuppacoffee.com/location/">Cuppacoffee</a> provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything &mdash; the order + napkins, milk, cream cheese, butter, and knives &mdash; ourselves because Cuppacoffee is very close to MTW.</p> + +<p><a href="http://www.cmartboston.com/">CMart</a> sold us water and juices. (There is also a Whole Foods near MTW.)</p> + +<p>At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour&rsquo;s <a href="https://youtu.be/kIbhckqanHI">sticky buns</a>.</p> + +<h3 id="event-program">Event Program</h3> + +<p>The following assumptions/goals constrained the schedule for the day:</p> + +<ul> + <li>give all 17 on-campus Ph.D. students a talk slot; talks must <strong>either</strong> communicate one technical idea from recent work <strong>or</strong> say what led the speaker to apply to a PL Ph.D. program</li> + <li>allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)</li> + <li>do not make the audience sit for more than 1 hour at a time</li> + <li>maximize the number and length of breaks (to encourage discussions)</li> + <li>include a relevant and fun welcome activity</li> + <li>save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions</li></ul> + +<p>We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.</p> + +<p>The activity was <em>codewalks</em>. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition/blob/master/src/main/java/com/seriouscompany/business/java/fizzbuzz/packagenamingpackage/impl/math/arithmetics/IntegerDivider.java">IntegerDivider</a> class from a silly FizzBuzz implementation and a solution to the <a href="https://en.wikipedia.org/wiki/Dutch_national_flag_problem">Dutch national flag problem</a>). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and <a href="https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/">egoless programming</a>.</p> + +<p>The talks ran conference-style. One organizer played &ldquo;session chair&rdquo; to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.</p> + +<p>For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.</p> + +<h2 id="looking-back-at-the-details">Looking Back at the Details</h2> + +<p>Reading Day in the Spring may be a good, free date for future retreats.</p> + +<p>We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.</p> + +<p>The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we&rsquo;d keep the morning breaks short &mdash; just enough to use the restroom and grab a coffee &mdash; and aim for 30-min breaks in the afternoon.</p> + +<p>Groups of three 15-minute talks worked well, but groups of four talks might be equally good.</p> + +<p>Perhaps each speaker should get the chance to pick a talk length. <a href="https://nepls.org/">NEPLS</a>, for example, allows a choice between 5-min. and 30-min. talks.</p> + +<p>The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick &ldquo;topics&rdquo; beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.</p> +<!-- to future organizers: the panel began asking faculty for a mistake in their career and wound up with a defensive flavor--> + +<h2 id="looking-back-overall">Looking Back, Overall</h2> + +<p>This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.</p> + +<p>There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time &mdash; keep the same date &amp; program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.</p> + +<p>With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.</p> + + Complete Monitors for Gradual Types + http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + Thu, 31 Oct 2019 21:58:26 UT + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + Forgetful and Heedful contracts + http://prl.ccs.neu.edu/blog/2019/04/07/forgetful-and-heedful-contracts/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-04-07-forgetful-and-heedful-contracts + Sun, 07 Apr 2019 23:15:11 UT + Ben Greenman + +<p><em>Forgetful</em> and <em>heedful</em> are two methods for space-efficient contracts developed by <a href="http://www.cs.pomona.edu/~michael/">Michael Greenberg</a> in <a href="https://arxiv.org/abs/1410.2813">2014</a>. These methods were born in the shadow of a third method, <em>eidetic</em>, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.</p> +<!-- more--> + +<hr /> + +<p>Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of <a href="https://www2.ccs.neu.edu/racket/pubs/popl11-dfff.pdf">three parties</a>: the API code, the client code, and the contract between them.</p> + +<p>For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:</p> + +<pre><code> #lang racket/base + + (module sort racket + (provide + (contract-out + [quicksort + (-&gt; (vectorof point/c) void?)])) + + (define point/c (vectorof integer?)) + + (define (quicksort points) + ....)) + + (module client racket + (require (submod ".." sort)) + (quicksort '())) + + (require 'client)</code></pre> + +<pre><code>quicksort: contract violation; + expected a vector + given: '() + in: the 1st argument of + (-&gt; (vectorof (vectorof integer?)) void?) + contract from: + (file.rkt sort) + blaming: (file.rkt client) + (assuming the contract is correct)</code></pre> + +<p>That covers the basics. For an extended introduction to contracts, visit <a href="https://docs.racket-lang.org/guide/contracts.html">The Racket Guide</a>.</p> + +<p>The quicksort example and the related figures are from the paper <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts: Fixing a Pathology of Gradual Typing</em></a></p> + +<h3 id="classic-contracts-and-space-efficiency">Classic contracts and &ldquo;Space Efficiency&rdquo;</h3> + +<p>The <code>(vectorof point/c)</code> contract used above describes a possibly-mutable array whose elements match the <code>point/c</code> contract. Since the array can be mutated, this contract has implications for two parties:</p> + +<ol> + <li>the client module must supply a good array, and</li> + <li>the sorting module must not insert a bad element.</li></ol> + +<p>To enforce the second condition, the <code>vectorof</code> contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:</p> + +<pre><code>(quicksort (vector (vector 4 4) + (vector 2 2) + (vector 1 1) + (vector 3 3)))</code></pre> + +<p>After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a <strong>solid</strong> blue bar for the <strong>write</strong> checks against the sort module and a <em>striped</em> blue bar for the <em>read</em> checks against the client.</p> + +<p><img src="/img/vector-chaperone-0.png" alt="A wrapped vector" /></p> + +<p>In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted &mdash; because every <strong>write</strong> and <em>read</em> applies a contract to the element.</p> + +<p><img src="/img/vector-chaperone-1.png" alt="Layers of element wrappers" /></p> + +<p>On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.</p> + +<p>Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and &ldquo;indirecting&rdquo; future writes/reads through wrappers. These space and time costs can add up.</p> + +<blockquote> + <p>&ldquo;on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times&rdquo; &mdash; <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts</em></a></p></blockquote> + +<p>To fix the problem, researchers have been exploring <em>space-efficient</em> implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and <em>eidetic</em>, <em>forgetful</em>, and <em>heedful</em> are his names for three implementations.</p> + +<p>(Although the goal of this post is to promote <em>forgetful</em> and <em>heedful</em>, we will review all three.)</p> + +<h3 id="eidetic-space-efficiency">Eidetic space-efficiency</h3> + +<p>The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a <em>merge</em> operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal &ldquo;wrapping&rdquo; contracts and their size is bounded by the number (and height) of source-code contracts in the program.</p> + +<p>An eidetic contract is an <code>N</code>-ary tree (for <code>N &gt; 0</code>):</p> + +<ul> + <li>each node represents a higher-order contract combinator, such as <code>vectorof</code></li> + <li>the <code>N</code> children of a node represent the different interactions that the value supports</li> + <li>each leaf is a list of non-higher-order, or <em>flat</em>, contracts</li></ul> + +<p>For example, the <code>(vectorof point/c)</code> source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf">Collapsible Contracts</a> paper has an illustration. Each tree node represents a <code>vectorof</code> contract; these nodes have <code>N=2</code> children because vectors support reads and writes.</p> + +<p>A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists &mdash; if an implementation comes with a <code>contract-stronger?</code> predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.</p> + +<p>Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should <code>contract-stronger?</code> try to solve problems that lack polynomial-time solutions?</p> + +<p>Thankfully, there are at least two &ldquo;compromise&rdquo; alternatives.</p> + +<h3 id="forgetful-space-efficiency">Forgetful space-efficiency</h3> +<!-- "no operation relies on e being a T2, skipping the check doesn't risk soundness" p.12--> +<!-- "In forgetful \lambda_H, we offer a simple solution to space inefficient casts: just forget about them" p.11--> +<!-- "Just the same, when accumulating casts on the stack, we throw away all but the last cast" p.11--> +<!-- "forgetful ... skip[s] checks and change[s] blame labels" p.3--> + +<blockquote> + <p>&ldquo;Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 10, bottom of page 23--></blockquote> + +<p>The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:</p> + +<ol> + <li>the client obligations from the old contract, and</li> + <li>the server obligations from the new contract</li></ol> + +<p>If not, proceed as usual &mdash; by wrapping (an unwrapped value) or raising an error. Every value receives at most <strong>one</strong> wrapper; this wrapper changes as the value flows to different clients.</p> + +<p>Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module <code>A</code> exports a function <code>f</code> with contract <code>(-&gt; T1 T2)</code> to module <code>B</code>, and suppose module <code>B</code> shares this function with a few other client modules using different contracts. As <code>f</code> flows to a new client, it keeps the <code>T1</code> domain check and gets a replacement for the <code>T2</code> codomain check.</p> + +<ul> + <li>Keeping <code>T1</code> ensures that the code inside the function (defined by module <code>A</code>) receives input that matches its expectation.</li> + <li>Replacing <code>T2</code> ensures that each new client receives output that it expects.</li></ul> + +<p>Unfortunately, replacing <code>T2</code> also means that clients of module <code>B</code> cannot trust the <code>T2</code> contract. This contract is not checked, and so forgetful contracts <strong>miss</strong> some errors that would be caught by standard contracts. For the same reason, a bug in module <code>B</code> may go undetected by its clients &mdash; even if a later contract reports an issue, the contract system has no memory that <code>B</code> was partly-responsible.</p> + +<p>Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.</p> + +<h3 id="heedful-space-efficiency">Heedful space-efficiency</h3> + +<p>A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set &mdash; or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.</p> + +<p>To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.</p> + +<p>The heedful method is a compromise between forgetful and eidetic.</p> + +<ul> + <li> + <p>Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of <code>contract-stronger?</code> predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.</p></li> + <li> + <p>Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.</p></li></ul> + +<p>For details, see <a href="https://arxiv.org/abs/1410.2813">the extended version</a> of Michael&rsquo;s POPL 2015 paper. Don&rsquo;t bother searching <a href="http://www.cs.pomona.edu/~michael/papers/popl2015_space.pdf">the conference version</a> &mdash; aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.</p> + +<p><code>*</code> If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to <a href="http://mballantyne.net/">Michael Ballantyne</a> for explaining this to me.</p> + +<h3 id="priorities-and-appearances">Priorities and Appearances</h3> + +<p>The extended version of <em>Space-Efficient Manifest Contracts</em> introduces the forgetful and heedful methods with extreme modesty. It&rsquo;s tempting to skip past them and focus on the eidetic method.</p> + +<blockquote> + <p>&ldquo;Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [<a href="https://dl.acm.org/citation.cfm?doid=1706299.1706342">Siek and Wadler 2010</a>]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen&mdash;warm up exercises for eidetic.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 1, bottom of page 2--></blockquote> + +<p>And yet, at least two other research papers rely on these &ldquo;strawmen&rdquo; &mdash; or rather, the ideas behind the names.</p> + +<p><a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a>, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.</p> + +<blockquote> + <p>&ldquo;if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- page 21--></blockquote> + +<p>This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.</p> + +<blockquote> + <p>&ldquo;removing these casts preserves the soundness of the evaluation while reducing the number of them&rdquo;</p> + <p>&ldquo;while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- p.27--><!-- page 21--></blockquote> +<!-- The followup at POPL 2019 is not forgetful.--> +<!-- It's similar to eager coercions ... keep all types around and error--> +<!-- if there's a new type that doesn't match the old ones.--> +<!-- Also, that paper chooses not to let functions have intersection types,--> +<!-- which kind-of-avoids the questions ... but really the eagerness is key.--> + +<p><a href="https://dl.acm.org/citation.cfm?id=3009849"><em>Big Types in Little Runtime</em></a>, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their <em>transient</em> semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.</p> + +<p>That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value&rsquo;s set. When a mismatch occurs, the semantics <a href="https://www.ccs.neu.edu/home/types/resources/notes/transient-undefined-blame-extract.pdf">tries to report</a> every type in the set that relates to the mismatch.</p> + +<p>And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">ICFP 2018</a>. Where will we see them next?</p> + + Writing a paper with Scribble + http://prl.ccs.neu.edu/blog/2019/02/17/writing-a-paper-with-scribble/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-02-17-writing-a-paper-with-scribble + Sun, 17 Feb 2019 16:20:50 UT + Ben Greenman + +<p>This post explains how to get started using Scribble to write a research paper.</p> +<!-- more--> + +<hr /> + +<blockquote> + <p>This post was written using <a href="http://download.racket-lang.org/all-versions.html">Racket 7.1</a> and <a href="https://github.com/racket/scribble/releases/tag/v7.1">Scribble 1.29</a></p></blockquote> + +<p>Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:</p> + +<ul> + <li>the paper can import definitions from the research, keeping a single point of control;</li> + <li>the language&rsquo;s functional abstractions can help manage the writing;</li> + <li>the language&rsquo;s drawing and/or plotting libraries can replace <a href="https://ctan.org/pkg/pgf?lang=en">TikZ</a>;</li> + <li>and you can write unit tests to validate the claims made in the paper.</li></ul> + +<p>Scribble, <a href="http://docs.racket-lang.org/scribble/index.html">the Racket documentation tool</a>, comes with a to-LaTeX compiler and a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> library tailored to the new <a href="https://ctan.org/pkg/acmart?lang=en">ACM paper format</a>. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short &ldquo;getting started&rdquo; guide and links to some larger examples.</p> + +<blockquote> + <p>For a Scribble tutorial, see the links in: <a href="/blog/2017/05/23/building-a-website-with-scribble/index.html">Building a Website with Scribble</a></p></blockquote> + +<h2 id="getting-started-with-">Getting started with <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a></h2> + +<p>The first line of a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> document sets the formatting options (similar to a LaTeX file using <code>acmart.cls</code>). For example, the <a href="https://conf.researchr.org/track/gpce-2018/gpce-2018#Call-for-Papers">GPCE 2018 call for papers</a> asks for anonymized <code>sigplan</code>-format submissions with line numbers and 10 point font. The proper Scribble incantation is:</p> + +<pre><code>#lang scribble/acmart @sigplan @anonymous @review @10pt</code></pre> + +<p>Next, you may want to import some definitions. If we have a file <code>references.rkt</code> (see below for a definition), we can import it as follows:</p> + +<pre><code>@require{references.rkt}</code></pre> + +<p>The third main ingredient is the title and author information:</p> + +<pre><code>@(define neu (affiliation #:institution "Northeastern University")) +@(define anon (email "anon@anon.net")) + +@title{Writing a paper with Scribble} +@author[#:affiliation neu #:email anon]{Ben Greenman} + +@; optional: set the author names in the page headers +@elem[#:style "Sshortauthors"]{B. Greenman}</code></pre> + +<p>The paper is now ready to be written. You can forge ahead with a new <a href="http://docs.racket-lang.org/scribble/base.html#%28def._%28%28lib._scribble%2Fbase..rkt%29._section%29%29">section</a> and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and <a href="http://docs.racket-lang.org/scribble/base.html#%28form._%28%28lib._scribble%2Fbase..rkt%29._include-section%29%29">import</a> content from other modules:</p> + +<pre><code>@include-abstract{abstract.scrbl} +@include-section{introduction.scrbl}</code></pre> + +<p>Finally, the main page is a good place to <a href="https://docs.racket-lang.org/scriblib/autobib.html">generate the bibliography</a>. Assuming this document imports a file like the <code>references.rkt</code> below, this expression inserts a bibliography titled &ldquo;References&rdquo;:</p> + +<pre><code>@generate-bibliography[#:sec-title "References"]</code></pre> + +<p>To build the document, invoke <code>scribble</code> on the command-line with the <code>--pdf</code> or <code>--latex</code> options:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl</code></pre> + +<p>If all goes well, this command generates a <code>FILE.pdf</code> with properly-linked cross references.</p> + +<h3 id="auxiliary-files">Auxiliary Files</h3> + +<p>If you save the code above to a file <code>example.scrbl</code> and save the files below in the same directory, then you should be able to build an <code>example.pdf</code>.</p> + +<p>These files are available in a slightly different format at this link:</p> + +<ul> + <li><a href="https://gitlab.com/bengreenman/scribble-acmart-example">https://gitlab.com/bengreenman/scribble-acmart-example</a></li></ul> + +<h4 id="referencesrkt"><code>references.rkt</code></h4> + +<pre><code>#lang racket/base + +(provide + ~cite citet generate-bibliography + fbf-icfp-2009) + +(require + scriblib/autobib) + +(define-cite ~cite citet generate-bibliography + #:style author+date-square-bracket-style) + +(define icfp "ICFP") + +(define fbf-icfp-2009 + (make-bib + #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools" + #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler") + #:location (proceedings-location icfp #:pages '(109 120)) + #:date 2017))</code></pre> + +<h4 id="abstractscrbl"><code>abstract.scrbl</code></h4> + +<pre><code>#lang scribble/acmart + +A simple Scribble document.</code></pre> + +<h4 id="introductionscrbl"><code>introduction.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require{references.rkt} + +@; start with `title` instead of `section`, because importing via +@; `include-section` shifts all title/section/subsections down one level +@title{Introduction} + +Scribble creates a connection between a stand-alone document and the artifact +it describes@~cite[fbf-icfp-2009].</code></pre> + +<h3 id="q-how-to-debug-scribble-error-messages">Q. How to debug Scribble error messages?</h3> + +<p>If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.</p> + +<p>As a compile-time example, adding <code>@ foo</code> to a document produces the message <code>unexpected whitespace after @</code> and you can either delete the whitespace or change the <code>@</code> to <code>@"@"</code> for a literal <code>@</code>-sign.</p> + +<p>As a run-time example, adding <code>@(+ 2 2)</code> produces this message:</p> + +<pre><code>not valid in document body (need a pre-part for decode) in: 4</code></pre> + +<p>One fix is to convert <code>4</code> to a string, as in <code>@~a[(+ 2 2)]</code>.</p> + +<p>But if something goes wrong when Scribble renders a generated document to PDF, the default error output is <strong>not</strong> likely to help. For example, adding <code>@elem[#:style "oops"]</code> to a document produces a giant message:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl +[[ ... 84K of output ... ]] +Output written on example.pdf (1 page, 277876 bytes). +PDF statistics: + 53 PDF objects out of 1000 (max. 8388607) + 37 compressed objects within 1 object stream + 7 named destinations out of 1000 (max. 500000) + 36877 words of extra memory for PDF output out of 42996 (max. 10000000) + +run-pdflatex: got error exit code + context...: + [[ ... 17 more lines ... ]]</code></pre> + +<p>The best way to debug these messages is to <strong>ignore them</strong> and use a LaTeX compiler directly. For the &ldquo;oops&rdquo; mistake, LaTeX stops at the undefined control sequence &mdash; giving a hint about how to find the problem:</p> + +<pre><code>$ raco scribble --latex FILE.scrbl +$ pdflatex FILE.tex +[[ ... 12KB of output ... ]] +! Undefined control sequence. +l.549 \oops + {} +? </code></pre> + +<h3 id="q-how-to-add-a-latex-style-file">Q. How to add a LaTeX style file?</h3> + +<p>To add extra LaTeX code to the final document, create a new file and include it with the <code>++style</code> command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).</p> + +<pre><code>$ raco scribble ++style style.tex --pdf FILE.scrbl</code></pre> + +<p>Here is an example style file.</p> + +<h4 id="styletex"><code>style.tex</code></h4> + +<pre><code>\settopmatter{printfolios=true,printccs=true,printacmref=true} +% add page numbers etc. + +\overfullrule=1mm +% draw a black rectangle near lines that overflow the margin</code></pre> + +<p>Another way to add extra LaTeX code is to add a <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Flatex-properties..rkt%29._tex-addition%29%29"><code>tex-addition</code></a> style property to the main title. This second approach makes it easy to include more than one file:</p> + +<pre><code>#lang scribble/acmart + +@require[ + (only-in scribble/core make-style) + (only-in scribble/latex-properties make-tex-addition)] + +@(define extra-style-files + (list (make-tex-addition "style.tex"))) + +@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble} + +@; ....</code></pre> + +<h3 id="q-how-to-make-a-figure">Q. How to make a figure?</h3> + +<p>Use the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library to add figures to a document.</p> + +<pre><code>@require[pict scriblib/figure] +@figure[ + "fig:fish" @; figure tag, see `figure-ref` + @elem{A Standard Fish} @; figure caption, appears below the content + @elem{fish = @(standard-fish 90 40)}] @; content</code></pre> + +<p>The content of a figure can be almost anything that would work in the toplevel of the document.</p> + +<h3 id="q-how-to-include-extra-files-pictures-latex">Q. How to include extra files (pictures, LaTeX)?</h3> + +<p>The <code>++extra</code> command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.</p> + +<p>For example, if a document includes the content of an external LaTeX file:</p> + +<pre><code>@elem[#:style "input"]{inline-this.tex}</code></pre> + +<p>then make sure to build the document with a command like this:</p> + +<pre><code>$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl</code></pre> + +<h4 id="inline-thistex"><code>inline-this.tex</code></h4> + +<pre><code>% Raw LaTeX allowed here +$\lambda x.\, x$</code></pre> + +<h3 id="q-what-about-in-line-latex">Q. What about in-line LaTeX?</h3> + +<p>An <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Fcore..rkt%29._element%29%29">element</a> with the <a href="https://docs.racket-lang.org/scribble/core.html#%28idx._%28gentag._60._%28lib._scribblings%2Fscribble%2Fscribble..scrbl%29%29%29"><code>'exact-chars</code></a> <a href="https://docs.racket-lang.org/scribble/core.html#%28tech._style._property%29">style property</a> renders directly to LaTeX.</p> + +<pre><code>@(define (exact . stuff) + @; the style name "relax" puts a `\relax` no-op in front of the stuff + (make-element (make-style "relax" '(exact-chars)) stuff)) + +@exact|{$\lambda x.\, x$}| +@; ==&gt; \relax{$\lambda x.\, x$} + +@(define ($ . math-stuff) + (apply exact (list "$" math-stuff "$"))) + +@${\lambda x.\, x} +@; ==&gt; \relax{$\lambda x.\, x$}</code></pre> + +<h2 id="creating-a-httpdocsracket-langorgguidemodulesyntaxhtml28parthash-lang29lang-for-a-paper">Creating a <a href="http://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29">#lang</a> for a paper</h2> + +<p>For a Scribble document that is split across multiple files, it can be helpful to make a <code>#lang</code> that <a href="http://blog.racket-lang.org/2017/03/languages-as-dotfiles.html">provides a common environment</a>. Instead of starting each file with a <code>require</code>, e.g.:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require["references.rkt" "helper-functions.rkt" scriblib/figure] + +....</code></pre> + +<p>files can start with a name that describes their common purpose:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang conference-2018-submission + +....</code></pre> + +<p>As a bonus, if the language is defined as a package then the Scribble document can use Racket&rsquo;s dependency management tools:</p> + +<pre><code># to install the paper and interactively install dependencies: +$ cd conference-2018-submission; +$ raco pkg install + +# To check that the paper builds with no dependency issues: +$ raco setup --check-pkg-deps conference-2018-submission + +# To run all unit tests +$ raco test -c conference-2018-submission</code></pre> + +<p>To create a package and language:</p> + +<ol> + <li>Move the Scribble document to a directory with the language name, i.e., <code>conference-2018-submission/</code></li> + <li>Write a simple <code>info.rkt</code> to configure the package</li> + <li>Create a normal Racket module that exports the common environment</li> + <li>Create a <code>conference-2018-submission/lang/reader.rkt</code> module</li></ol> + +<p>Details below. For a full example, visit:</p> + +<ul> + <li><a href="https://gitlab.com/bennn/scribble-acmart-example">https://gitlab.com/bennn/scribble-acmart-example</a></li></ul> + +<hr /> + +<h4 id="conference-2018-submissioninforkt"><code>conference-2018-submission/info.rkt</code></h4> + +<p>This file defines the basic metadata for a package. For more about <code>info.rkt</code>, see: <a href="http://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html">Tutorial: Creating a Package</a>.</p> + +<pre><code>#lang info +(define collection "conference-2018-submission") +(define deps '("base" "scribble-lib" "at-exp-lib")) +(define build-deps '("racket-doc" "scribble-doc")) +(define pkg-desc "Paper for Conference 2018") +(define version "0.1")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionmainrkt"><code>conference-2018-submission/main.rkt</code></h4> + +<p>This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> language, the file &ldquo;references.rkt&rdquo;, and the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library.</p> + +<pre><code>#lang racket/base + +(provide + (all-from-out + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")) + +(require + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionlangreaderrkt"><code>conference-2018-submission/lang/reader.rkt</code></h4> + +<p>This file: (1) tells Racket to use the Scribble reader on <code>#lang conference-2018-submission</code> modules, and (2) wraps the result of such modules in a shape that Scribble expects.</p> + +<pre><code>#lang s-exp scribble/base/reader +conference-2018-submission +#:wrapper1 (lambda (t) (cons 'doc (t)))</code></pre> + +<h2 id="links-to-example-documents">Links to Example Documents</h2> + +<p>These documents use the <code>#lang</code> approach to writing a paper with Scribble. Check their <code>main.rkt</code> for example formatting functions and unit tests, and check the <code>.scrbl</code> files to see how the ideas above look in a larger document.</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018">https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018</a></li> + <li><a href="https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018">https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018</a></li></ul> + +<p>Finally, this repository provides a tool to start a new Scribble document:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gtp-paper">https://pkgd.racket-lang.org/pkgn/package/gtp-paper</a></li></ul> + +<h2 id="further-reading">Further Reading</h2> + +<ul> + <li><a href="https://project.inria.fr/coqexchange/checking-machine-checked-proofs/">Checking Machine-Checked Proofs</a></li></ul> + + The Behavior of Gradual Types: A User Study + http://prl.ccs.neu.edu/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + Tue, 11 Dec 2018 19:50:33 UT + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + Java and Migratory Typing + http://prl.ccs.neu.edu/blog/2018/12/02/java-and-migratory-typing/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + Sun, 02 Dec 2018 14:41:53 UT + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> + + Disappearing Code + http://prl.ccs.neu.edu/blog/2018/11/24/disappearing-code/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-11-24-disappearing-code + Sat, 24 Nov 2018 09:52:58 UT + Ben Greenman + +<p>Two experiences at <a href="https://2018.splashcon.org/home">SPLASH 2018</a> reminded me that software gets thrown away and replaced.</p> +<!-- more--> + +<h3 id="story-1">Story 1</h3> + +<p>The first reminder came near the end of a <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a> by <a href="https://people.csail.mit.edu/rinard/">Martin Rinard</a>. Once upon a time, Martin was working as a consultant and a firm asked him to review a software package. (The firm wanted a second opinion about how the software computed its results.) The firm sent a zipfile; Martin found six versions of the code inside; the firm said &ldquo;well, please check all six versions&rdquo;; and it turned out:</p> + +<ul> + <li><strong>Version 1</strong> : the source code was written in a domain-specific language (DSL) that generated code for the application</li> + <li><strong>Version 2</strong> : the DSL source was the same as version 1, but the generated code was slightly modified</li> + <li>&hellip;</li> + <li><strong>Version 6</strong> : the generated code was the source code and the DSL was gone</li></ul> + +<p>The moral of Martin&rsquo;s story was: (1) the creators of a software system are often different from the maintainers, and (2) researchers need to build tools to help these maintainers.</p> + +<h3 id="story-2">Story 2</h3> + +<p>The second reminder came from a teaching assistant who said the <a href="https://www.cs.cornell.edu/courses/cs3110/2018fa/">functional programming course</a> at their institution was currently using a Python script to test students&rsquo; code. Once upon a time, I was a teaching assistant for the <a href="https://www.cs.cornell.edu/courses/cs3110/2014sp/">same course</a> at the same institution. We had trouble testing students&rsquo; code via the Python script left by the pre&ndash;2013 course staff, so I wrote a <a href="https://gitlab.com/bengreenman/ocaml_tools/">command-line tool</a> to handle the tests and other compile/run/grade tasks. To keep history from repeating itself, I used the same language the course teaches (OCaml) and wrote some documentation &mdash; but it seems like that was not enough. At any rate, writing the tool was a good exercise.</p> + +<blockquote> + <p><em>In the end, everybody must understand for himself.</em> &mdash; <a href="https://dl.acm.org/citation.cfm?id=3731">Per Martin-Löf</a></p></blockquote> + +<h3 id="reflection">Reflection</h3> + +<p>In each story, the maintainers of a software system threw away some old code to make their job easier in the short term. How can we stop this &ldquo;re-inventing the wheel&rdquo; from happening?</p> + +<p>Martin Rinard&rsquo;s solution is to let maintenance programmers keep their current habits, but provide tools to make the short-term, pragmatic solutions into a more robust systems. Search for "<a href="https://people.csail.mit.edu/rinard/paper/osdi04.pdf">failure-oblivious computing</a>" to learn more (this was the topic of his <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a>).</p> + +<p>In Story 1, the maintainers were able to avoid the DSL by modifying an inherited blob of DSL-generated code. If the DSL did not generate code, history might have taken a different course; it might be best to start with a language that offers tools for linguistic re-use, and to build a DSL from these tools &mdash; so there is no generated code. The Racket programming language is exploring this path. For a recent example, see the <a href="https://www2.ccs.neu.edu/racket/pubs/icfp17-acf.pdf">video-lang paper</a>.</p> + +<p>The Story 2 test harness, however, was not generating code. Its maintainers discarded a &ldquo;big&rdquo; program written in a typed functional language in favor of a script. Perhaps we need a language that allows mixing statically-typed and dynamically-typed code (shouts out to <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">my own research</a>).</p> + +<p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p> + +<hr /> + +<h4 id="addendum-comment-from-mitch-wand">Addendum: comment from Mitch Wand</h4> + +<blockquote> + <p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p></blockquote> + +<p>Ermm, this works better for sourdough bread than for people.</p> + +<p>Even in the not-so-real world of checking student solutions, there&rsquo;s often no way of guaranteeing that one half of a pair will be around for the second round. They may be on co-op. Or the course will not be offered the next semster/year/etc. Or the course will change at the next offering (from OCaml to Python or from Racket to Java) so that large chunks of the infrastructure will have to be discarded or rewritten.</p> + +<p>The &ldquo;real&rdquo; solution is to write literate code (as we preached incessantly in PDP), so that the next reader will have at least some clue as about what you wrote. This just may be sufficient incentive to modify rather than rebuild from scratch.</p> + +<p>Ever the optimist, &mdash;Mitch</p> + + A Spectrum of Type Soundness and Performance + http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-06-a-spectrum-of-type-soundness-and-performance + Sat, 06 Oct 2018 11:23:35 UT + Ben Greenman + +<p>The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>A Spectrum of Type Soundness and Performance</em> by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018">http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018</a></p></blockquote> + +<p>A dynamically-typed language runs any program that &ldquo;looks good&rdquo; (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both &ldquo;looks good&rdquo; and is well-typed according to a type checker.</p> + +<p>A <em>mixed-typed</em> language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.</p> + +<div class="figure"><img src="/img/mixed-typed-systems-by-year.png" alt="Figure 1: Some mixed-typed languages" /> + <p class="caption">Figure 1: Some mixed-typed languages</p></div> + +<p>These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:</p> + +<ul> + <li><strong>MACLISP</strong> defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the <a href="http://www.softwarepreservation.org/projects/LISP/MIT/Moon-MACLISP_Reference_Manual-Apr_08_1974.pdf">Moonual</a>. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).</li> + <li><strong>Strongtalk</strong> includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a> position paper.</li> + <li><strong>Typed Racket</strong> lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.</li> + <li><strong>Thorn</strong> requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.</li> + <li><strong>Reticulated</strong> lets a program combine static and dynamic <em>expressions</em> and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek <a href="https://dl.acm.org/citation.cfm?id=3009849">POPL 2017</a>).</li></ul> + +<p>That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system&rsquo;s protocol for checking dynamically-typed values at a <em>type boundary</em> (between static and dynamic code).</p> + +<h3 id="main-contribution">Main Contribution</h3> + +<p>In the paper <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>A Spectrum of Type Soundness and Performance</em></a>, we define a tiny mixed-typed language and show three ways to define the behavior of programs &mdash; based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.</p> + +<p>The three behaviors are inspired by existing languages. A <strong>higher order</strong> behavior ensures that dynamically-typed values match the static type at a boundary &mdash; by checking the value when possible, and by monitoring the value&rsquo;s future interactions when necessary. A <strong>first order</strong> behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An <strong>erasure</strong> behavior does no checking whatsoever.</p> + +<blockquote> + <p>Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function <code>f</code>, then one way to enforce the type boundary is to wrap <code>f</code> in a proxy to assert that every future call to <code>f</code> returns a number. In this case, the proxy monitors the behavior of <code>f</code>.</p></blockquote> + +<p>Concretely, the paper defines three formal semantics with the same names. The <strong>higher-order</strong> semantics enforces full types at the boundaries (Section 2.3). The <strong>first-order</strong> semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every &ldquo;selector&rdquo; operation in typed code, e.g., function application, data structure access (Section 2.5). The <strong>erasure</strong> semantics simply ignores the types (Section 2.4).</p> + +<p>Each semantics satisfies a <em>different</em> notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">online supplement</a> gives full proofs.</p> + +<p>The paper has more to say about the meta-theory. See section 2 and section 4.</p> + +<blockquote> + <p>To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for <a href="https://dl.acm.org/citation.cfm?id=2676971">subset of the language</a> (in the spirit of <a href="http://soundiness.org/">soundiness</a>) or use the name &ldquo;type soundness&rdquo; to describe <a href="https://dl.acm.org/citation.cfm?id=2676971">a different property</a>.</p></blockquote> + +<p>Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations &mdash; similar to Typed Racket&rsquo;s <a href="http://docs.racket-lang.org/ts-reference/Typed_Racket_Syntax_Without_Type_Checking.html">no-check</a> language.</p> + +<p>Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (<a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016</a>). The programs we measured are the non-object-oriented ones from our <a href="http://docs.racket-lang.org/gtp-benchmarks/index.html">benchmark suite</a>.</p> + +<p>To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises &mdash; see section 3 of the paper, section B of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, and/or the <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slides</a>.</p> + +<h3 id="implications">Implications</h3> + +<ol> + <li>The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, or <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slide 114</a>.</li> + <li>Programmers using mixed-typed languages need to know what guarantees their types provide. (It is <a href="https://twitter.com/jbandi/status/965005464638541825">not safe to assume that TypeScript types give the same guarantees as OCaml types</a>!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.</li> + <li>The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?</li></ol> + +<blockquote> + <p>Note: in this post, a <em>mixed-typed language</em> is one that allows any combination of static and dynamic typing. A <em>gradually-typed language</em> is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>).</p></blockquote> + + Sampling Gradual Typing Performance + http://prl.ccs.neu.edu/blog/2018/05/08/sampling-gradual-typing-performance/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + Tue, 08 May 2018 15:37:37 UT + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> + + The Racket School 2018: Create your own language + http://prl.ccs.neu.edu/blog/2018/04/27/the-racket-school-2018-create-your-own-language/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-04-27-the-racket-school-2018-create-your-own-language + Fri, 27 Apr 2018 21:35:22 UT + Ben Greenman + +<p>The Racket School 2018: Create your own language • 9–13 July • Salt Lake City</p> +<!-- more--> + +<p>The Racket team has spent over thirty years developing and refining a coherent intellectual tradition for studying and building programming languages. This year’s school will introduce participants to Racket’s framework for language-oriented programming, which the summer school faculty recently spelled out in a a cover article in the <a href="https://tinyurl.com/RacketCACM">Communications of the ACM</a></p> + +<p>Concretely, the 2018 Racket Summer School will cover the following topics:</p> + +<ul> + <li>the spectrum of programming languages;</li> + <li>modules and syntax, or languages as libraries;</li> + <li>DrRacket’s support for language-oriented programming;</li> + <li>a domain-specific language for adding types to languages;</li> + <li>tools and techniques for implementing notational conveniences; and</li> + <li>research challenges in language-oriented programming.</li></ul> + +<p>If these topics intrigue you, attend the Racket Summer School:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2018/">http://summer-school.racket-lang.org/2018/</a></li></ul> + +<p>This is not your run-of-the-mill summer school. We will do our best to make it exciting, entertaining, and useful to a broad spectrum of attendees, both academic and industrial.</p> + +<p>P.S. We will send you your first problem set in June, a month before the summer school to whet your appetite.</p> + + PLT Redex FAQ + http://prl.ccs.neu.edu/blog/2017/09/25/plt-redex-faq/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-25-plt-redex-faq + Mon, 25 Sep 2017 23:39:16 UT + Ben Greenman, Sam Caldwell + +<p>A short guide to Redex concepts, conventions, and common mistakes.</p> +<!-- more--> + +<hr /> + +<p><em>To contribute to this FAQ, submit issues and pull requests to: <a href="https://github.com/nuprl/website/">https://github.com/nuprl/website/</a></em></p> + +<h3 id="q-what-is-redex-useful-for">Q. What is Redex useful for?</h3> + +<ol> + <li>declaring <a href="https://en.wikipedia.org/wiki/Regular_tree_grammar">regular tree grammars</a></li> + <li>defining <em>pattern</em>-based judgments and relations on <em>terms</em></li> + <li>testing properties of the above</li></ol> + +<p>More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.</p> + +<h3 id="q-what-is-redex-not-useful-for">Q. What is Redex <strong>not</strong> useful for?</h3> + +<p>Proving theorems about a grammar, judgment, or relation.</p> + +<h3 id="q-what-is-a-term">Q. What is a <em>term</em>?</h3> + +<p>Informally, a term is:</p> + +<ul> + <li>a Redex &ldquo;atom&rdquo;, or</li> + <li>an object that represents a sequence of characters.</li></ul> + +<p>More formally, a term is the result of evaluating <strong>(term X)</strong>, where <strong>X</strong> is any syntactically-correct Racket expression.</p> + +<p>Examples:</p> + +<pre><code>$ racket +Welcome to Racket v6.10.0.3. +&gt; (require redex/reduction-semantics) +&gt; (term 42) +42 +&gt; (term (+ 2 2)) +'(+ 2 2) +&gt; (term ("hello" world (#false))) +'("hello" world (#f))</code></pre> + +<p>Some terms may look strange. That&rsquo;s OK, because a term by itself has no meaning.</p> + +<p>Terms can refer to previously-defined values using the <strong>unquote</strong> escape.</p> + +<pre><code>&gt; (define x (term 42)) +&gt; (term (+ 2 x)) +'(+ 2 x) +&gt; (term (+ ,x (unquote x))) +'(+ 42 42)</code></pre> + +<h3 id="q-what-is-a-redex-model">Q. What is a <em>Redex model</em>?</h3> + +<p>A Redex model is collection of tools for working with terms. The tools may include:</p> + +<ul> + <li><em>languages</em>, to define a grammar for terms</li> + <li><em>judgments</em>, to describe properties of terms or relations between terms</li> + <li><em>metafunctions</em>, to transform terms</li> + <li><em>reduction relations</em>, to define a term rewriting system</li></ul> + +<p>The goal of these tools is to encode a &ldquo;real thing&rdquo; (maybe, a programming language) using Redex terms.</p> + +<h3 id="q-what-is-a-language">Q. What is a language?</h3> + +<p>A Redex <em>language</em> is a named set of non-terminals, <em>patterns</em>, and <em>binding forms</em>. For example, a Redex model of the natural numbers might start with this language:</p> + +<pre><code>(define-language nat + [N ::= Zero + (Plus1 N)])</code></pre> + +<ul> + <li>the name of the language is <strong>nat</strong></li> + <li>the non-terminal <strong>N</strong> is associated with two patterns: <strong>Zero</strong> and <strong>(Plus1 N)</strong></li> + <li>there are no <em>binding forms</em></li></ul> + +<p>Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.</p> + +<p>The non-terminal <strong>N</strong> describes all terms that are either:</p> + +<ol> + <li>the symbol <strong>Zero</strong></li> + <li>lists of the form <strong>(Plus1 N)</strong>, where <strong>N</strong> is either <strong>Zero</strong> or another &ldquo;Plus1&rdquo;</li></ol> + +<p>For example,</p> + +<pre><code>(term Zero) +(term (Plus1 Zero)) +(term (Plus1 (Plus1 Zero))) +(term (Plus1 (Plus1 (Plus1 Zero)))) +;; .... and so on</code></pre> + +<p>If a language has binding forms, then some terms can introduce names. See the question on <em>binding forms</em> (below) for an example.</p> + +<h3 id="q-what-is-a-pattern">Q. What is a pattern?</h3> + +<p>A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains <em>named non-terminals</em> from the language, then you can ask whether a Redex term matches the pattern.</p> + +<p>A <em>named non-terminal</em> for a language <strong>L</strong> is an identifier made of: (1) a non-terminal from <strong>L</strong>, (2) an underscore (<strong>_</strong>), and (3) any other identifier. See the FAQ entry below.</p> + +<p>For example, <strong>(redex-match? L p t)</strong> returns <strong>#true</strong> if the term <strong>t</strong> matches the pattern <strong>p</strong> relative to the language <strong>L</strong>.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(redex-match? nat N_some-name (term Zero)) +;; #true +(redex-match? nat (Plus1 N_a) (term Zero)) +;; #false +(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; #true</code></pre> + +<p>If <strong>(redex-match? L p t)</strong> is <strong>#true</strong>, then <strong>(redex-match L p t)</strong> shows how named non-terminals in the pattern bind to subterms of <strong>t</strong>.</p> + +<pre><code>(redex-match nat N_0 (term Zero)) +;; (list (match (list (bind 'N_0 'Zero)))) +(redex-match nat (Plus1 N_0) (term Zero)) +;; #f +(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))</code></pre> + +<h3 id="q-what-is-a-named-non-terminal">Q. What is a named non-terminal?</h3> + +<p>A named non-terminal in a language <strong>L</strong> is an identifier of the form <strong>X_Y</strong>, where:</p> + +<ul> + <li><strong>X</strong> is a non-terminal from <strong>L</strong></li> + <li><strong>Y</strong> is any identifier</li></ul> + +<p>The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.</p> + +<pre><code>(define-language trees + [binary-tree ::= Leaf + (Node binary-tree binary-tree)]) + +(redex-match trees + (Node binary-tree_left binary-tree_right) + (term (Node Leaf (Node Leaf Leaf)))) +;; (list +;; (match +;; (list (bind 'binary-tree_left 'Leaf) +;; (bind 'binary-tree_right '(Node Leaf Leaf)))))</code></pre> + +<h3 id="q-what-else-can-patterns-express">Q. What else can patterns express?</h3> + +<p>Redex patterns may contain special identifiers to guide pattern-matching. For instance:</p> + +<ul> + <li>The <strong>_</strong> pattern matches any term (and does not bind).</li> + <li>A pattern <strong>(p &hellip;)</strong> matches any sequence whose elements match the pattern <strong>p</strong>. If the pattern <strong>p</strong> is a named non-terminal, then the non-terminal binds a sequence of subterms.</li></ul> + +<p>Examples:</p> + +<pre><code>(redex-match? nat (Plus1 _) (term (Plus1 9))) +;; #true +(redex-match? nat (N_0 ...) (term ())) +;; #true +(redex-match? nat (N_0 ...) (term (Zero))) +;; #true +(redex-match nat (N_0 ...) (term (Zero Zero Zero))) +;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))</code></pre> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29">the Redex reference</a> for the full pattern language, including the <em>named and unique non-terminals</em> of the form <strong>X_!_Y</strong>.</p> + +<h3 id="q-what-can-patterns-not-express">Q. What can patterns <strong>not</strong> express?</h3> + +<ul> + <li>Disjunctions of patterns, e.g., &ldquo;number or boolean&rdquo;. (Use a language non-terminal.)</li> + <li>Negations of patterns. (Compose <strong>not</strong> with <strong>redex-match?</strong>.)</li> + <li>Some non-regular patterns. (Named dots <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29"><code>..._N</code></a> or <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-language%29%29"><code>define-language</code></a> with a side condition may be able to help.)</li></ul> + +<h3 id="q-what-is-a-judgment">Q. What is a judgment?</h3> + +<p>A Redex <em>judgment</em> form defines a relation on terms. The relation is defined by a set of inference rules.</p> + +<p>Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:</p> + +<ul> + <li>well-formedness conditions (i.e., whether a term contains free variables)</li> + <li>type checking rules</li> + <li>type inference rules</li> + <li>evaluation relations</li></ul> + +<p>Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(define-judgment-form nat + #:mode (N= I I) + #:contract (N= N N) + [ + --- Zero= + (N= Zero Zero)] + [ + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1) + (N= N_0-- N_1--) + --- Plus1= + (N= N_0 N_1)])</code></pre> + +<ol> + <li>the language is <strong>nat</strong>; Redex uses the language to interpret patterns</li> + <li>the mode is <strong>(N= I I)</strong>; this means <strong>N=</strong> is the name of a judgment that expects two input terms (or, <strong>N=</strong> is a binary relation on terms)</li> + <li>the contract is <strong>(N= N N)</strong>; in other words, <strong>N=</strong> expects two terms that match the <strong>N</strong> non-terminal from the <strong>nat</strong> language</li> + <li>there are two inference rules, named <strong>Zero=</strong> and <strong>Plus1=</strong></li> + <li>the <strong>Zero=</strong> rule states that <strong>(N= Zero Zero)</strong> always holds</li> + <li>the <strong>Plus1=</strong> rule states that <strong>(N= N_0 N_1)</strong> holds if <strong>N_0</strong> and <strong>N_1</strong> are both <strong>Plus1</strong> terms whose contents are related by <strong>N=</strong></li></ol> + +<p>The <strong>where</strong> clauses are <em>guards</em>. When Redex tries to apply a rule with premises of the form <strong>(where pattern term)</strong>, it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-judgment-form%29%29">the Redex reference</a> for more.</p> + +<pre><code>(judgment-holds (N= Zero Zero)) +;; #true +(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Note: the inference rules form a <em>set</em>, not a <em>sequence</em>. So when you ask Redex whether <strong>(judgment-holds (N= Zero Zero))</strong>, it applies all rules that match <strong>(N= Zero Zero)</strong>. For <strong>N=</strong> this is just one rule, but in general it could be many rules.</p> + +<h3 id="q-what-is-a-judgment-form-mode">Q. What is a judgment form <strong>#:mode</strong>?</h3> + +<p>A <strong>#:mode</strong> declaration expects a list of the form <strong>(id pos-use &hellip;)</strong>, where <strong>id</strong> is an identifier and each <strong>pos-use</strong> is either <strong>I</strong> or <strong>O</strong>. These declarations say four things:</p> + +<ol> + <li><strong>id</strong> is the name of a new judgment form</li> + <li><strong>id</strong> expects <strong>N</strong> arguments, where <strong>N</strong> is the number of <strong>pos-use</strong> symbols</li> + <li><strong>id</strong> expects an <em>input</em> at each position where the mode contains an <strong>I</strong></li> + <li><strong>id</strong> produces an <em>output</em> at each position where the mode contains an <strong>O</strong></li></ol> + +<p>For example, a type inference judgment may take an expression as input and output a type. Here&rsquo;s a fast and easy type inference judgment for arithmetic expressions. Given any term <strong>e_0</strong>, the judgment outputs the type <strong>Int</strong>.</p> + +<pre><code>(define-language Arith + (e ::= integer (e + e)) + (τ ::= Int)) + +(define-judgment-form Arith + #:mode (infer-type I O) + #:contract (infer-type e τ) + [ + --- T-Int + (infer-type e_0 Int)])</code></pre> + +<h3 id="q-what-can-judgments-not-express">Q. What can judgments <strong>not</strong> express?</h3> + +<p>Redex does not support inference rules that require guessing.</p> + +<p>One example of this is a transitivity rule: "<strong>τ_0</strong> is related to <strong>τ_2</strong> if there exists a <strong>τ_1</strong> such that <strong>τ_0</strong> is related to <strong>τ_1</strong> and <strong>τ_1</strong> is related to <strong>τ_2</strong>". The following example tries to define a transitive subtyping (<strong>&lt;:</strong>) relation, but Redex rejects the <strong>S-Trans</strong> rule.</p> + +<pre><code>(define-language SomeTypes + (τ ::= (→ τ τ) Integer)) + +(define-judgment-form SomeTypes + #:mode (&lt;: I I) + #:contract (&lt;: τ τ) + [ + (&lt;: τ_0 τ_1) + (&lt;: τ_1 τ_2) + --- S-Trans + (&lt;: τ_0 τ_2)] + [ + --- S-Refl + (&lt;: τ_0 τ_0)] + [ + (&lt;: τ_dom-1 τ_dom-0) + (&lt;: τ_cod-0 τ_cod-1) + --- S-Arrow + (&lt;: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])</code></pre> + +<p>The error is that in the rule <strong>S-Trans</strong>, the named non-terminal <strong>τ_1</strong> appears in an input position but is not bound to a term.</p> + +<h3 id="q-what-is-a-metafunction">Q. What is a metafunction?</h3> + +<p>A metafunction is a term-level function on terms.</p> + +<p>Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.</p> + +<p>Here is a metafunction that returns <strong>#true</strong> when given two equal natural numbers. The definition is similar to the <strong>N=</strong> judgment form.</p> + +<pre><code>(define-metafunction nat + N=? : N N -&gt; boolean + [(N=? Zero Zero) + #true] + [(N=? N_0 N_1) + (N=? N_0-- N_1--) + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1)] + [(N=? N_0 N_1) + #false])</code></pre> + +<ul> + <li>the metafunction is named <strong>N=?</strong></li> + <li>its contract is <strong>N N -&gt; boolean</strong>, this means <strong>N=?</strong> expects 2 terms that match the <strong>N</strong> pattern and returns a term that matches the pattern <strong>boolean</strong></li> + <li>there are three cases; the second case is guarded by two <strong>where</strong> clauses</li></ul> + +<p>Any occurrence of <strong>(N=? &hellip;.)</strong> in any term is evaluated.</p> + +<pre><code>(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(term ((N=? Zero Zero) Zero)) +;; '(#true Zero) +(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Any occurrence of <strong>N=?</strong> outside a <strong>term</strong> is an error.</p> + +<p>Metafunction <strong>where</strong>-clauses are analogous to judgment form <strong>where</strong>-clauses. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._metafunction%29">the Redex reference</a> for more.</p> + +<p>Note: the cases in a metafunction form a <em>sequence</em>, not a <em>set</em>. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.</p> + +<h3 id="q-should-i-use-a-metafunction-or-a-judgment-form">Q. Should I use a metafunction or a judgment form?</h3> + +<p>Use a judgment form.</p> + +<p>Metafunctions are handy, but judgments are easier to read and debug and maintain.</p> + +<h3 id="q-what-is-a-reduction-relation">Q. What is a reduction relation?</h3> + +<p>A reduction relation is a set of term-rewriting rules.</p> + +<p>Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.</p> + +<ul> + <li>The language tells Redex how to interpret patterns.</li> + <li>The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.</li> + <li>The rewrite rules have the form <strong>(&mdash;&gt; term term guard &hellip;)</strong>. The term on the left is the input, the term on the right is the output.</li></ul> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._reduction-relation%29%29">the Redex reference</a> for a full description of the guards.</p> + +<p>The preferred way to define a reduction relation is to define a language that includes three non-terminals:</p> + +<ol> + <li>a non-terminal for the domain of the reduction relation</li> + <li>a non-terminal for a <em>subset</em> of the domain that cannot reduce further</li> + <li>a non-terminal for evaluation contexts</li></ol> + +<p>An evaluation context is a term that contains a <strong>hole</strong>. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite &mdash; in particular, the sub-term that matches the <strong>hole</strong>.</p> + +<p>In the following example, <strong>bexp</strong> is the domain of a reduction relation. A <strong>bexp</strong> term represents a boolean expression, which can be <strong>#true</strong> or <strong>#false</strong> or a conjunction of expressions or a disjunction of expressions. The boolean expressions <strong>#true</strong> and <strong>#false</strong> are also values (<strong>val</strong>); these cannot reduce further. The non-terminal <strong>E</strong> is for evaluation contexts.</p> + +<pre><code>(define-language Bool + (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp)) + (val ::= #true #false) + (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E))) + +(define step + (reduction-relation Bool + #:domain bexp + [--&gt; (in-hole E (val_lhs ∧ val_rhs)) + (in-hole E val_new) + ∧-step + (where val_new ,(and (term val_lhs) (term val_rhs)))] + [--&gt; (in-hole E (val_lhs ∨ val_rhs)) + (in-hole E val_new) + ∨-step + (where val_new ,(or (term val_lhs) (term val_rhs)))]))</code></pre> + +<p>The function <strong>apply-reduction-relation</strong> applies a reduction relation to a term and returns a list of ways that the term can step.</p> + +<pre><code>(apply-reduction-relation step (term #true)) +;; '() +(apply-reduction-relation step (term (#true ∧ #true))) +;; '(#true) +(apply-reduction-relation step (term (#true ∧ #false))) +;; '(#false) +(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true))) +;; '((#true ∧ #true))</code></pre> + +<p>Three things about the reduction relation <strong>step</strong>:</p> + +<ol> + <li>Using <strong>in-hole</strong> on the first argument of <strong>&mdash;&gt;</strong> searches a term for a subterm that Redex can apply a reduction rule to.</li> + <li>Using <strong>in-hole</strong> on the second argument of <strong>&mdash;&gt;</strong> puts a new value back into the <strong>hole</strong> in the evaluation context.</li> + <li>The unquote operator (<strong>,</strong>) escapes to &ldquo;Racket mode&rdquo; (see below) to evaluate a conjunction or disjunction.</li></ol> + +<p>A judgment or metafunction is a formal alternative to &ldquo;escaping to Racket&rdquo;, but escaping can be convenient.</p> + +<p>Note: the cases in a reduction relation form a <em>set</em>, not a <em>sequence</em>. If more than one case matches, Redex applies them all.</p> + +<h3 id="q-what-is-racket-mode-what-is-redex-mode">Q. What is &ldquo;Racket mode&rdquo;? What is &ldquo;Redex mode&rdquo;?</h3> + +<p>Code in a Redex model is sometimes evaluated in &ldquo;Racket mode&rdquo; and sometimes evaluated in &ldquo;Redex mode&rdquo;. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.</p> + +<p>Key points:</p> + +<ol> + <li>A Redex program starts in Racket mode.</li> + <li>The <strong>X</strong> in <strong>(term X)</strong> is evaluated in Redex mode &hellip;</li> + <li>&hellip; unless <strong>X</strong> contains unquoted sub-expressions. Unquoting escapes to Racket mode &hellip;</li> + <li>&hellip; and <strong>term</strong>s inside an unquoted sub-expression are evaluated in Redex mode.</li></ol> + +<p>In other words, <strong>term</strong> enters Redex mode and <strong>unquote</strong> (<strong>,</strong>) escapes back to Racket.</p> + +<p>Redex implicitly switches to Redex mode in a few other places, for example:</p> + +<ul> + <li>the right-side of a <strong>where</strong> clause is in Redex mode</li> + <li>the result of a metafunction is in Redex mode</li></ul> + +<p>When in doubt, try using an <strong>unquote</strong>. Redex will raise an exception if it finds an unquote in Racket mode.</p> + +<h3 id="q-are-side-conditions-evaluated-in-racket-mode-or-redex-mode">Q. Are <strong>side-condition</strong>s evaluated in &ldquo;Racket mode&rdquo; or &ldquo;Redex mode&rdquo;?</h3> + +<p>A <strong>(side-condition e)</strong> sometimes evaluates <strong>e</strong> as a Racket expression and sometimes evaluates <strong>e</strong> as a Redex expression.</p> + +<ul> + <li>reduction relations and metafunctions expect a <strong>Racket</strong> expression</li> + <li>judgments expect a <strong>Redex</strong> expression</li></ul> + +<h3 id="q-what-is-a-binding-form">Q. What is a binding form?</h3> + +<p>In the lambda calculus, <strong>λ</strong>-terms bind variables. A term <strong>(λ x M)</strong> means that any free occurrence of <strong>x</strong> in the sub-term <strong>M</strong> refers to the <strong>x</strong> from the <strong>λ</strong>-term.</p> + +<p>Redex can express this idea with a binding form.</p> + +<pre><code>(define-language Λ + [e ::= (e e) x (λ x e)] + [x ::= variable-not-otherwise-mentioned] + #:binding-forms + (λ x_0 e_0 #:refers-to x_0))</code></pre> + +<p>Note: all the non-terminals in a language must be defined before the <strong>#:binding-forms</strong> keyword. If a non-terminal definition appears after the <strong>#:binding-forms</strong> keyword, then Redex will interpret the &ldquo;definition&rdquo; as a binding form.</p> + +<p>Binding forms work together with Redex&rsquo;s functions for substitution and alphabetic equivalence.</p> + +<pre><code>(alpha-equivalent? Λ + (term (λ x x)) + (term (λ y y)))) +;; #true + +(define-metafunction Λ + test-substitute : e -&gt; e + [(test-substitute (λ x_0 e_0)) + (substitute e_0 x_0 y)]) +(term (test-substitute (λ z (z z)))) +;; '(y y)</code></pre> + +<h3 id="q-what-is--what-is-">Q. What is <strong>&hellip;</strong>? What is <strong>&hellip;.</strong>?</h3> + +<p>Three dots (<strong>&hellip;</strong>) is for building patterns. If <strong>p</strong> is a pattern then <strong>(p &hellip;)</strong> matches any list whose elements all match <strong>p</strong>.</p> + +<pre><code>(define-language L) +(redex-match? L (number ... boolean ...) (term (1 2 #true #true))) +;; #true</code></pre> + +<p>Four dots (<strong>&hellip;.</strong>) may be used in <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-extended-language%29%29"><strong>define-extended-language</strong></a> to extend a previously-defined non-terminal.</p> + +<pre><code>(define-language C + (keyword ::= auto break case)) +(define-extended-language C++ + C + (keyword ::= .... class)) + +(redex-match? C keyword (term auto)) +;; #true +(redex-match? C keyword (term class)) +;; #false +(redex-match? C++ keyword (term auto)) +;; #true +(redex-match? C++ keyword (term class)) +;; #true</code></pre> + +<h3 id="q-where-to-learn-more-about-redex">Q. Where to learn more about Redex?</h3> + +<p>&ldquo;Critical path&rdquo; resources:</p> + +<ul> + <li>Code examples for this post: <a href="https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt">https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt</a></li> + <li>Redex documentation: <a href="http://docs.racket-lang.org/redex/index.html">http://docs.racket-lang.org/redex/index.html</a></li> + <li><a href="https://dvanhorn.github.io/redex-aam-tutorial/"><em>An Introduction to Redex with Abstracting Abstract Machines</em></a> by David Van Horn</li> + <li><a href="https://www.leafac.com/playing-the-game-with-plt-redex/"><em>Playing the Game with PLT Redex</em></a> by Leandro Facchinetti</li> + <li><a href="https://williamjbowman.com/doc/experimenting-with-redex/index.html"><em>Experimenting with Languages in Redex</em></a> by William J. Bowman</li></ul> + +<p>&ldquo;Procrastination&rdquo; resources:</p> + +<ul> + <li><a href="http://tata.gforge.inria.fr/"><em>Tree Automata Techniques and Applications</em></a></li> + <li><a href="http://lamport.azurewebsites.net/pubs/lamport-types.pdf"><em>Should your Specification Language be Typed?</em></a></li> + <li>Redex source code (see <code>redex-lib/</code>): <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + + Gradual Typing Across the Spectrum, part II + http://prl.ccs.neu.edu/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-22-gradual-typing-across-the-spectrum-part-ii + Tue, 22 Aug 2017 15:54:06 UT + Ben Greenman + +<p>Last week, Northeastern hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:</p> + +<p><a href="http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html">http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html</a></p> + +<p>A common thread among the talks was the question: <em>how to convert a research idea into a tool for software developers?</em></p> +<!-- more--> + +<p>In my mind, gradual typing <em>is</em> an answer to one instance of this question. The research idea is strong static type systems, and the software developers are the millions using dynamically typed languages. I know that static typing can make programs easier to write and maintain. The developers know that dynamic typing has benefits; moreover they know better than to migrate their code from one language to another on a whim. Gradual typing is a linguistic solution to the problem of <em>adding</em> the benefits of static typing to a dynamically typed language.</p> + +<p>Enough opinions, let&rsquo;s talk about the talks.</p> + +<p>The morning session consisted of four talks:</p> + +<ul> + <li> + <p><a href="https://www.cs.umd.edu/people/milod">Milod Kazerounian</a> (<a href="https://www.cs.umd.edu/">UMD</a>) spoke about upgrading the <a href="https://github.com/plum-umd/rdl">RDL</a> type checker for Ruby with support for refinement types. The idea is to compile Ruby code and types to <a href="https://emina.github.io/rosette/">Rosette</a>, and profit from <a href="http://yices.csl.sri.com/papers/cav2007.pdf">SMT</a>-assisted type checking.</p></li> + <li> + <p><a href="http://ambrosebs.com/">Ambrose Bonnaire-Sergeant</a> (<a href="https://www.cs.indiana.edu/">IU</a>, <a href="http://ambrosebs.com/talks/squash-work-boston-pi-2017.pdf">slides</a>) has been inferring <em>useful</em> <a href="http://typedclojure.org/">Typed Clojure</a> types through dynamic analysis of Clojure programs. His tool observes how values flow through a program at run-time, then lifts these observations into possibly-recursive, possibly-incorrect type annotations. The surprising result is that the tool quickly (1&ndash;2 seconds per unit test, I think) infers types that can help a developer start annotating a program.</p></li> + <li> + <p><a href="http://ccs.neu.edu/~types/">Ben Greenman</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>, <a href="http://homedirs.ccs.neu.edu/types/resources/talks/preservation-types.pdf">slides</a>) explained why he is implementing a semantics for <a href="https://github.com/racket/typed-racket">Typed Racket</a> inspired by Michael Vitousek&rsquo;s work on <a href="http://homes.soic.indiana.edu/mvitouse/papers/popl17.pdf">Reticulated Python</a>. The &ldquo;why&rdquo; is &ldquo;performance&rdquo;. The Reticulated semantics will enforce a notion of tag soundness in kind of <a href="https://en.wikipedia.org/wiki/Deal_with_the_Devil">devils contract</a> to improve performance.</p></li> + <li> + <p><a href="https://cs.brown.edu/~ptunnell/">Preston Tunnell-Wilson</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tpk-crowdsource-lang-design/">ONWARD 2017</a>) recently sent questions about programming language design to <a href="https://www.mturk.com/mturk/welcome">Mechanical Turk</a> workers. Survey says, developers have extremely diverse opinions about what they <em>expect</em> and what they <em>want</em> regarding scope, inheritance, and infix operators.</p></li></ul> + +<p>In the early afternoon, we had two talks on similar themes as the morning session:</p> + +<ul> + <li> + <p><a href="https://github.com/akuhlens">Andre Kuhlenschmidt</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is exploring the design space of efficient implementations for run-time type checks. The main challenge is how to <em>monitor</em> higher-order data in a way that efficiently performs type checks and can help the programmer debug any failed checks. This talk presented data comparing two approaches to the program; I believe the latter, improved approach is based on <a href="http://homepages.inf.ed.ac.uk/wadler/papers/coercions/coercions.pdf">coercions</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) explained that there are many ways to adapt type soundness to a gradually typed language, and presented some data comparing Typed Racket&rsquo;s <em>generalized soudness</em> to Reticulated Python&rsquo;s <em>tag soundness</em>. The data suggests that tag soundness never adds an order-of-magnitude slowdown.</p></li></ul> + +<p>Next on the schedule were two talks about implementing advanced type systems in Racket&rsquo;s macro expander (think: meta-level linguistic re-use, capture-avoiding substitution for free!)</p> + +<ul> + <li> + <p><a href="https://github.com/iitalics">Milo Turner</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) first showed how to implement <a href="https://gankro.github.io/blah/linear-rust/#definitions-and-the-state-of-rust">linear and affine</a> type systems using <a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html">syntax-parse</a>, and second presented a simpler implementation using the <a href="http://docs.racket-lang.org/turnstile/index.html">Turnstile</a> library.</p></li> + <li> + <p><a href="http://www.davidchristiansen.dk/">David Christiansen</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is building <a href="https://github.com/david-christiansen/pudding">a proof assistant</a> in Racket. This talk focused on the design and implementation of proof tactics.</p></li></ul> + +<p>After a short break, we heard about something completely different:</p> + +<ul> + <li><a href="http://justinpombrio.net/">Justin Pombrio</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/research/plt/dl/icfp2017/">ICFP 2017</a>) taught us to interpet the scoping rules of a &ldquo;core&rdquo; language as a preorder. Using the preorder, he then showed how to <em>infer</em> the scoping rules of any &ldquo;surface&rdquo; language based on its translation to the &ldquo;core&rdquo;.</li></ul> + +<p>Last summer and fall, Jeremy Siek hosted two REUs (<a href="https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5517&amp;from=fund">research experience for undergraduates</a>) at Indiana University. The two students gave the next talks:</p> + +<ul> + <li> + <p>Di Zhong (<a href="https://www.cs.indiana.edu/">IU</a>) talked about implementing interpreters in Racket, Python, and Haskell. As I understand, this was a hands-on experience through <a href="https://www.cis.upenn.edu/~bcpierce/tapl/">TAPL</a> and <a href="https://redex.racket-lang.org/">the Redex book</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="https://www.cs.indiana.edu/">IU</a>) demonstrated her implementation of <a href="http://theory.stanford.edu/~aiken/publications/papers/popl94.pdf">conditional types</a> for <a href="https://github.com/mvitousek/reticulated">Reticulated</a>.</p></li></ul> + +<p>Finally,</p> + +<ul> + <li><a href="https://nikivazou.github.io/">Niki Vazou</a> (<a href="https://www.cs.umd.edu/">UMD</a>) presented a theory of gradual refinement types. Any &ldquo;holes&rdquo; in the refinements introduce a search problem; type checking attempts to solve the problem by finding a predicate that unifies a function definition and its callers.</li></ul> + +<p>This meeting was a great opportunity to reflect on the recent past and share opinions on what&rsquo;s worth pursuing in the future. Many thanks to the participants, and to the NSF for the support!</p> + +<blockquote> + <p>If you want to know about the future, you need to ask the young people who will create it. Young people don&rsquo;t know what can&rsquo;t be done, and so they go ahead and do it. &mdash; <a href="https://www.youtube.com/watch?v=sM1bNR4DmhU">Ivan Sutherland</a></p></blockquote> + + Trees, 1973 + http://prl.ccs.neu.edu/blog/2017/07/19/trees-1973/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-07-19-trees-1973 + Wed, 19 Jul 2017 21:48:56 UT + Ben Greenman + +<p>From the PRL archives:</p> + +<blockquote> + <p>I think that I shall never see a matrix lovely as a tree. &mdash; <a href="/img/gls-trees-poem-1979.pdf"><em>Trees</em></a>, by Guy L. Steele Jr., MIT, 1973</p></blockquote> +<!-- more--> + +<hr /> + +<p>You might recognize the opening line from Joyce Kilmer&rsquo;s 1914 poem <a href="https://en.wikipedia.org/wiki/Trees_(poem)"><em>Trees</em></a>, or from Radia Perlman&rsquo;s <a href="/img/p-sigcomm-1985.pdf"><em>Algorhyme</em></a> (published 1985).</p> + +<p>The poem is online in <a href="http://mercury.lcs.mit.edu/~jnc/humour/lisp.tree">at least one other place</a>, but the copy linked above (from <a href="https://archive.org/details/byte-magazine">BYTE magazine</a>) comes with a footnote on <em>How this poem came to be printed</em>.</p> + + Continuations + http://prl.ccs.neu.edu/blog/2017/07/17/continuations/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-07-17-continuations + Mon, 17 Jul 2017 12:52:07 UT + Ben Greenman + +<p>From the PRL archives:</p> + +<blockquote> + <p>It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. &mdash; <a href="/img/nall-continuations-1983.pdf"><em>Continuations</em></a> by Alan Nall, Indiana University, 1983</p></blockquote> +<!-- more--> + +<hr /> + +<p>I first encountered this essay on continuations in a green folder in the PRL. It turns out, the author spent a semester at Indiana University working on the <a href="http://wiki.c2.com/?SameFringeProblem">same fringe problem</a> for a graduate-level programming languages course. According to <a href="https://www.cs.indiana.edu/~dfried/">the instructor</a>: &ldquo;What he said was true. He could not stop thinking about the problem the entire semester.&rdquo; This essay was a kind of final exam.</p> + + Quotes and Stories from "Turing 50" + http://prl.ccs.neu.edu/blog/2017/06/24/quotes-and-stories-from-turing-50/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-24-quotes-and-stories-from-turing-50 + Sat, 24 Jun 2017 20:00:52 UT + Ben Greenman + +<p>The ACM recently hosted <a href="https://www.acm.org/turing-award-50">a celebration of 50 years of the A.M. Turing award</a>. These are some notes and thoughts from the event, including how Fred Brooks once rented a bus, Don Knuth&rsquo;s outrageous implementation of batch processing, and Judea Pearl&rsquo;s theory of homo sapiens.</p> +<!-- more--> + +<script>document.createElement('dialog');</script> + +<p><strong>Conventions / Disclaimers:</strong></p> + +<ul> + <li> + <p>The blockquotes below are paraphrased, may be incorrect, and may be incorrectly attributed. Make sure to watch the ACM&rsquo;s live stream before quoting anything here!!!</p></li> + <li> + <p>Section-breaks are labeled as &ldquo;panel&rdquo;, &ldquo;talk&rdquo;, &ldquo;question&rdquo;, etc.</p></li> + <li> + <p>This is intentionally &ldquo;bad writing&rdquo; in the Peter Lee sense (see below) &mdash; primarily &ldquo;what I saw&rdquo;, very little about &ldquo;what I thought and felt&rdquo;. A summary in my own words just wouldn&rsquo;t do justice to the panelists.</p></li> + <li> + <p>The &ldquo;Augmented Reality&rdquo; session was my favorite.</p></li></ul> + +<h3 id="opening-remarks">Opening Remarks</h3> + +<h4 id="alan-turing-is-with-us-today"><em>Alan Turing is with us today</em></h4> + +<p>At the start of the event, the <a href="http://users.ecs.soton.ac.uk/wh/">emcee</a> unveiled a bronze bust of Alan Turing. This statue was on display at center stage during the whole event.</p> + +<p>It&rsquo;s a good sculpture and it&rsquo;s good we remember Alan Turing, but I&rsquo;m sad that the ACM would encourage this kind of idol-worship. Let&rsquo;s not forget Turing&rsquo;s excellent teachers and colleagues!</p> + +<h3 id="talk-impact-of-turing-recipients-work">talk: Impact of Turing Recipients&rsquo; Work</h3> + +<h5 id="barbara-liskov">Barbara Liskov</h5> + +<blockquote> + <p><em>the first awards recognized achievements in the standard fields of theory, AI, and systems</em></p> + <p><em>hostile environment around the first awards, trepidation about future awards</em></p> + <p><em>with Unix, Ritchie and Thompson got the design right</em></p> + <p><em>Niklaus Wirth: &ldquo;If I understood how important Pcode was, I would have spent more time designing it&rdquo;</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>What is &ldquo;systems&rdquo; &mdash; does that even have a definition? And Unix is definitely NOT an example of a &ldquo;right design&rdquo;; rather it&rsquo;s a landmark of <a href="https://www.dreamsongs.com/WorseIsBetter.html">worse is better</a> design.</p> + +<h3 id="panel-advances-in-deep-neural-networks">panel: Advances in Deep Neural Networks</h3> + +<h4 id="stuart-russell">Stuart Russell</h4> + +<blockquote> + <p> <em>I work in all areas of AI except for deep learning</em></p></blockquote> + +<h4 id="judea-pearl">Judea Pearl</h4> + +<blockquote> + <p><em>I am a foreigner in this field &hellip; left because human beings are not good at handling information &hellip; people are very good with causal inference, not with statistical inference &hellip; deep learning is statistical</em></p> + <p><em>there is a very old existence proof, homo sapiens took over the planet &hellip; I believe because they had an internal model of their environment &hellip; a drawing of a lion with wings is evidence of this model, you have to have such a model before you can experiment with it and imagine &hellip; snakes have superb optics, result of a long evolution process &hellip; very specific but they cannot build eyeglasses &hellip; humans have an internal model, can build a market based on promises and build large communities based on promises</em></p> + <p><em>I see four levels &hellip; second level is predicting events, if I do X then what? &hellip; third level is counterfactual, if I did things differently then how would the outcome change &hellip; very hard to advance between levels, are we working to help machine learning &lsquo;level up&rsquo;?</em></p> + <p><em>data science is about the relation between data and reality &hellip; data alone is not data science</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>today we can&rsquo;t think without holding a piece of metal</em></p> + <p><em>machine learning is part of computer science rather than AI &hellip; AI is about how to make human &hellip; machine learning is about allocating resources &hellip; matrices are not all of human intelligence &hellip; neural nets are part of a wider toolbox &hellip; too much hype in NLP its just syntax</em></p> + <p><em>huge gap between syntax and semantics &hellip; chat bots are just syntax, don&rsquo;t learn &hellip; faking intelligence with neural nets, so well that you can build a company &hellip;</em></p> + <p><em>real metric is task completion</em></p> + <p><em>if I say &lsquo;a GLEEB walked across the airport&rsquo; then true intelligence can make a lot of educated guesses about a &lsquo;GLEEB&rsquo; without any other context</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>I disagree, ML is part of AI &hellip; understanding intelligence and making intelligent methods for solving AI problems</em></p> + <p><em>to quote Churchhill &lsquo;its not beginning of end, not end, not beginning of end, probably end of beginning&rsquo;</em></p> + <p><em>todays AI powered by hardware and data</em></p> + <p><em>AI cannot yet find our keys</em></p> + <p><em>quote: &lsquo;todays AI is making a perfect chess move while the world is on fire&rsquo; &hellip; ignores context</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>Turing &hellip; a program is a mathematical object &hellip; math community did not recognize this</em></p> + <p><em>lots of grad student descent &hellip; tuning to get performance &hellip; deep learning is neglecting the problem of exponential data &hellip; deep learning is just circuits, circuits lack expressive power &hellip; a human can process data from CERN but a neural net cannot, need to know physics</em></p> + <p><em>probabilistic programming, somewhat under the radar, maybe on the right track &hellip; 10-line program running/generating/updating a large network of possibilities &hellip; more composable and flexible</em></p></blockquote> + +<h5 id="ilya-sutskever">Ilya Sutskever</h5> + +<blockquote> + <p><em>why I like deep learning &hellip; philosophically satisfying &hellip; the hypothesis class is a circuit &hellip; powerful hypothesis class not too many parameters &hellip; can actually find circuits &hellip; &lsquo;violates all theory&rsquo; &hellip; really amazing &hellip; humans can see and hear pretty fast, even though our neurons are pretty slow, perhaps because we do a massively parallel process that doesn&rsquo;t take many steps &hellip; works well enough to be useful</em></p> + <p><em>models e.g. for vision are very hard to understand &hellip; fight fire with fire &hellip; incomprehensible solution to incomprehensible problem</em></p></blockquote> + +<h5 id="raquel-urtasun">Raquel Urtasun</h5> + +<blockquote> + <p><em>the breakthrough in neural nets is not algorithms &hellip; it is tricks, hardware, and grad students</em></p> + <p><em>with neural nets we forget about modeling, uncertainty, and prior knowledge &hellip; perception is a canonical example</em></p></blockquote> + +<h4 id="question-boundaries">question: boundaries</h4> + +<h5 id="judea-pearl">Judea Pearl:</h5> + +<blockquote> + <p><em>glad to see people in deep learning understand its limitations &hellip; is there a clearer definition of the boundaries? Are you worried about bridging the levels factual/inferential/counterfactural?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>the big problem is decision making under uncertainty</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>cognition is a hard problem</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>do you have a clear idea of the boundaries?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>neural nets use back-propagation &hellip; its non-modular, sad fact &hellip; performance and explainability is the tradeoff &hellip; then again people are non-modular</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>AlphaGo is not deep learning &hellip; basically an improved version of the machines Arthur Samuel made in the late 1950s &hellip; the interesting code is in C++ &hellip; rules of go, next moves, searching future states &hellip; depends on transitive closure</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>can AlphaGo take advice from a human?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>not currently, but that would be a new policy to add to the toolbox &hellip; just as neural nets are one tool within AlphaGo</em></p></blockquote> + +<h5 id="raquel-urtasun">Raquel Urtasun</h5> + +<blockquote> + <p><em>no reason to ask if deep learning is going to solve all problems</em></p></blockquote> + +<h4 id="question-education">question: education?</h4> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>indeed, what DO you teach in your neural networks classes?</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>&hellip; chain rule, Taylor expansion</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>teaching is communicating truths &hellip; what is true about neural nets? what are some things that will definitely not happen?</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>Peter Norvig and I have a problem with our AI book &hellip; chapter on vision, chapter on speech, will probably post just point to the neural nets chapter &hellip; we don&rsquo;t really understand! &hellip; really selling students short</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>in labs we talk about what we cannot do &hellip; we all have open problems</em></p> + <p><em>Stuart I hope you have a very good author for the chapters. There are so many open problems to communicate to students!</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>CS cirriculum needs more statistics, inferential thinking &hellip; revise the whole cirriculum bottom-up to weave this in</em></p></blockquote> + +<h4 id="question-could-a-neural-net-fix-my-phone-without-breaking-it">question: could a neural net fix my phone without breaking it?</h4> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>right! big problem that neural nets have no internal model to manipulate</em></p></blockquote> + +<h4 id="question-generalizability">question: generalizability?</h4> + +<h5 id="ilya-sutskever">Ilya Sutskever</h5> + +<blockquote> + <p><em>special-purpose vs. general purpose solution depends on the problem &hellip; most things we give special-purpose solutions &hellip; I guess if you wanted to automate a mathematician that would need to be general</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>always argue with your self &hellip; try to break what you&rsquo;ve built &hellip; there&rsquo;s a system that plays video games just using the pixels on screen as hints &hellip; it&rsquo;s very good at mazes; if a newborn baby learned to play maze games in 2 hours that would be amazing! &hellip; does the system scale? absolutely not</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>When Michael Jordan said &ldquo;people are non-modular&rdquo;, I think he means that people are able to break abstraction barriers when needed.</p> + +<h3 id="panel-restoring-personal-privacy-without-compromising-national-security">panel: Restoring Personal Privacy without Compromising National Security</h3> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>&hellip; wikileaks &hellip; russian hackers &hellip; social emergency &hellip;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>everything I say today is copyleft</em></p> + <p><em>its a misunderstanding to talk about a conflict between security and privacy &hellip; two aspects &hellip; problem goes back to feudalism &hellip; the right to build a castle was granted by the king &hellip; on one hand a castle improves national security &hellip; on the other hand a castle can be used to attack the king &hellip; technology is upsetting the basic notion of private vs. public security &hellip; governments cannot protect citizens and cannot protect themselves &hellip; extremely difficult to prove that a small process is secure</em></p> + <p><em>exceptional access makes it more complex</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>major concern are national security threats and ability of authorities to confound threats &hellip; analogy to printing press &hellip; proclimation of 1635 that only state messengers can carry letters &hellip; 1663 treatise by the national censor, no printing house can have a back door &hellip; the general topic is very old &hellip; title of this session isn&rsquo;t very good, the real dilemma is investigation vs privacy</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>code is law for better or worse, tech is not a tool like a watch &hellip; tech can monitor us and decide when it works &hellip; tech is government, not obedient tools &hellip; the mind is a warrant-proof space &hellip; 5th amendment rights should extend to wearables</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>cannot divorce the security/privacy issues from the current political context &hellip; the serious vulnerabilities are not in math &hellip; they are in users and implementors</em></p></blockquote> + +<h4 id="question-back-doors">question: back doors</h4> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>perhaps we should explain what a back door is</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>agency keeps a master key in escrow</em></p> + <p><em>non-lawyers can and should take a stand on basic issues</em></p> + <p><em>there are legitimate warrant-proof spaces &hellip; electronic extensions of the mind need to be recognized as warrant-proof spaces</em></p> + <p><em>the set of authorities with backdoor access should change as I travel between countries &hellip; but this will lead to a global race to the bottom</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>germany has a law against sex tourism (committed by German citizens visiting other countries) &hellip; neither government will be willing to lose backdoor access</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>technical reasons against backdoors &hellip; (1) &lsquo;weak crypto&rsquo; was implemented, nobody turned it off, is now breakable by anyone in 2015 &hellip; (2) Juniper used non-default crypto parameters, someone (inside?) changed the parameters &hellip; (3) attackers exploit back doors</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>quote &lsquo;you can put a man on the moon, surely you can put a man on the sun&rsquo;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>trouble is getting him back safely</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>I think back doors are okay, but not for personal devices &hellip; need public lab and transparent processes, need separation of powers &hellip; prosecutors are getting cases thrown out because courts do not accept their backdoors &hellip; there is a place for transparent back door tools</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>politicians are rarely technical people</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>tech is not a set of policy-neutral tools, need to address gap of understanding</em></p></blockquote> + +<h4 id="question-">question: ???</h4> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>we don&rsquo;t know how to build good crypto programs &hellip; opponents are debugging our programs with different goals &hellip; we&rsquo;re trying for-all-paths safety (universal) &hellip; they&rsquo;re trying exists-bad-path (existential)</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>cybersecurity market is a lemon market</em></p></blockquote> + +<h4 id="question-how-to-advise">question: how to advise</h4> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>question from audience &lsquo;I am an advisor to a company working with nuclear energy, they are terrified of being attacked, how should I advise them?&rsquo;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>a network like that is probably separated enough to be safe &hellip; the problem is being safe AND connected to the web</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>because the internet of things</em></p></blockquote> + +<h4 id="question-what-should-the-acm-do">question: what should the ACM do?</h4> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>maybe we need increased regulation, the ACM could help bring experts together</em></p></blockquote> + +<h4 id="question-what-is-true-security">question: what is true security</h4> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>it&rsquo;s all the same thing &hellip; gets labeled differently &hellip; just trying to control which bits can go where and who gets to read them</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>security is the absense of being violated</em></p></blockquote> + +<h5 id="paul-syverson-no-true--security-need-to-consider-context">Paul Syverson: <em>no true &gt; security, need to consider context</em></h5> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>problem of our community, have strict standards, may be unrealistic &hellip; maybe a lot more tolerance in practice than our model accepts</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>security and privacy are environmental problems</em></p></blockquote> + +<h4 id="question-can-we-stop-the-needle-in-haystack-search-for-vulnerabilities">question: can we stop the needle-in-haystack search for vulnerabilities?</h4> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>need to build in security from the start</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>need rule of law, transparency, separation of powers</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>stop delaying, instead of spending $$$ on fixing problems, we should invest in solving the fundamental issues</em></p></blockquote> + +<h3 id="panel-preserving-our-past-for-the-future">panel: Preserving our Past for the Future</h3> + +<p>Note: I was volunteering during this session; quotes are sparse</p> + +<h5 id="mahadev-satyanarayanan">Mahadev Satyanarayanan</h5> + +<blockquote> + <p><em>the running system is the total documentation &hellip; there are too many details for prose to capture</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>running old code has a danger of running old bugs</em></p></blockquote> + +<h5 id="">??:</h5> + +<blockquote> + <p><em>what not to save? &hellip; it&rsquo;s very hard to tell in advance</em></p></blockquote> + +<h5 id="mahadev-satyanarayanan">Mahadev Satyanarayanan:</h5> + +<blockquote> + <p><em>there is no absolute censor in a world with caching</em></p></blockquote> + +<h5 id="brewster-kahle">Brewster Kahle</h5> + +<blockquote> + <p><em>asking UNESCO to solve the problem is unrealistic &hellip; need to empower the fanatics, given them tools to preserve data</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>I totally agree with the &ldquo;empower the fanatics&rdquo; sentiment. Today, because of &ldquo;volunteer librarians&rdquo;, I think we&rsquo;re doing pretty well about preserving the past. Suppose I found an old PowerPoint file. I&rsquo;m sure I could find a way to read it with help from the internet &mdash; either by searching Google, pirating an old version of PowerPoint, or asking online forums. So personally I&rsquo;m not worried about losing data we have currently; I&rsquo;m more worried about the future, the internet becoming &ldquo;less chaotic&rdquo;.</p> + +<p>The panel raised a good question about how to preserve research and encourage reproducibility. A <code>.pdf</code> or <code>.tex</code> document is not enough; a virtual machine is okay. Really I think we need a stronger cultural emphasis on literate programming and a mature library like TeX to help authors store and share their work. <a href="https://thegamma.net/">The Gamma</a> seems on the right track.</p> + +<p>I was surprised that the panel did not discuss search, version control, and the ACM&rsquo;s open-access policy.</p> + +<h3 id="panel-moores-law-is-really-dead-whats-next">panel: Moore&rsquo;s Law is Really Dead: What&rsquo;s Next?</h3> + +<p>Note: I was volunteering during this session</p> + +<h5 id="butler-lampson">Butler Lampson</h5> + +<blockquote> + <p><em>there&rsquo;s plenty of room at the top &hellip; with Moore&rsquo;s Law we got improvements at the bottom of the software stack, everything above got to benefit and it was easy to integrate the changes &hellip; there&rsquo;s lots of opportunities to trim fat in the middle/top of the software stack &hellip; these improvements will be harder to integrate, but there&rsquo;s lots of opportunities</em></p></blockquote> + +<h5 id="margaret-martonosi">Margaret Martonosi</h5> + +<blockquote> + <p><em>By the way, don&rsquo;t believe the brochure that says I&rsquo;m at Google. My affiliation is Princeton, Google and I are just friends.</em></p></blockquote> + +<h5 id="butler-lampson">Butler Lampson</h5> + +<blockquote> + <p><em>important to distinguish approximate vs. precise software &hellip; precise software has a specification and the customer cares about that specification &hellip; approximate software doesn&rsquo;t have a hard spec, just needs to approximately work &hellip; the web is approximate, it doesn&rsquo;t work and it doesn&rsquo;t need to! &hellip; windows is precise, definitely has a spec and users definitely care</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>The recording of this panel should be good; it was very lively, very practical. And the first audience question (by <a href="https://people.eecs.berkeley.edu/~pattrsn/">David Patterson</a>) was &ldquo;an A+ question&rdquo;.</p> + +<p>The panel reminded me of a talk by <a href="http://users.ece.utexas.edu/~patt/">Yale Patt</a> about &ldquo;the end&rdquo; of the Von Neumann architecture. His opinion is future computers will be Von Neumann machines that rely on &ldquo;accelerators&rdquo; like a GPU &mdash; computer organization is not going to change, but will expand to have a bigger toolbox. So sure, Moore&rsquo;s Law is dead, but there are many opportunities to make computers faster at places other than the bottom of the software stack.</p> + +<h3 id="panel-challenges-in-ethics-and-computing">panel: Challenges in Ethics and Computing</h3> + +<p>Note: I was volunteering during this session</p> + +<h5 id="raj-reddy">Raj Reddy</h5> + +<blockquote> + <p><em>there are more slaves in the world currently than there were in the US during the civil war &hellip; here is one way technology could help, by giving everone a device to record their location &hellip; if someone&rsquo;s time and location is constant, they may be held against their will</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>do you believe every problem has a technological solution?</em></p></blockquote> + +<h5 id="noel-sharkey">Noel Sharkey</h5> + +<blockquote> + <p><em>yes the training set may be biased against people similar to me, but I want you to consider my case as an individual</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>a very nice Washington Post article</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>whether to encrypt the back hall</em></p></blockquote> + +<h5 id="raj-reddy">Raj Reddy</h5> + +<blockquote> + <p><em>we can sit here and wring our hands, but nothing will come of it unless it is written in the US constitution</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>I did not enjoy this panel. This is an ACM event, not a United Nations event. An ACM-sponsored panel about social and political problems should look for constructive ways that computer science can address these problems. Raj Reddy tried to give constructive solutions, but the panel seemed more interested in complaining about how hopeless things are.</p> + +<p>The comment by Noel Sharkey about &ldquo;consider me as an individual&rdquo; was something I hadn&rsquo;t thought about. Instead of worrying about biased datasets, let&rsquo;s use technology to collect data on an individual instead of abstracting a person by their race, age, or neighborhood.</p> + +<h3 id="talk-computer-science-as-a-major-body-of-accumulated-knowledge">talk: Computer Science as a Major Body of Accumulated Knowledge</h3> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p> <em>don&rsquo;t applaud me, just read my books</em></p> + <p><em>at the time, computer science was AI, numerical analysis, and programming languages</em></p> + <p><em>a colleague said &lsquo;I will believe that computer science is a science when it has 1000 deep theorems&rsquo; &hellip; I am not sure what a deep theorem is but I think its different from what&rsquo;s proven by deep learning</em></p> + <p><em>great privilege that we can invent the problems we work on &hellip; imagination &hellip; physicists can only guess the size of the sun</em></p> + <p><em>I&rsquo;ve always believed computer science and math are two parallel subjects &hellip; sometimes you hear people wondering if one subsumes the other</em></p> + <p><em>when I won the Turing Award, the prize money was about $1,000,000 less than it is today &hellip; I did get a nice Tiffany bowl that my wife and I use to serve strawberries &hellip; strawberries actually taste better &hellip;</em></p> + <p><em>very fortunate in this field &hellip; I&rsquo;m completely worthless as an economic advisor &hellip; it&rsquo;s a game I&rsquo;ve been able to take advantage of</em></p></blockquote> + +<h4 id="question-how-could-you-offer-to-pay-for-tex-bug-reports">question: how could you offer to pay for TeX bug reports?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>well there were many, many bugs &hellip; I stopped doubling at 32768 &hellip; brought people out of nowhere &hellip; next time I check the bug reports will be 2021 &hellip; someone is queueing the bugs reports &hellip; I believe strongly in batch rather than swap-in/swap-out &hellip; last time I checked reports was 2014 so 2021 will be next</em></p> + <p><em>TeX was a literate program, and it helped that I wrote &lsquo;The Errors of TeX&rsquo; about the first N bugs</em></p></blockquote> + +<h4 id="question-do-you-think-computers-will-become-good-composers-of-music-do-you-see-a-role-for-computer-assisted-proving">question: do you think computers will become good composers of music? do you see a role for computer-assisted proving?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>Yes in part, assisted is the key word &hellip; I have a program running now that I hope will help me prove a theorem</em></p></blockquote> + +<h4 id="question-favorite-algorithm">question: favorite algorithm?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>Tarjan&rsquo;s strong components &hellip; short deep useful</em></p></blockquote> + +<h4 id="question-thoughts-on-ai-computers-taking-over">question: thoughts on AI, computers taking over?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>I get scared when I see Stuart Russell making assumptions based on people acting rationally &hellip; then you look at election results</em></p></blockquote> + +<h4 id="question-if-you-could-start-over-and-do-things-differently-what-would-you-change">question: if you could start over and do things differently, what would you change?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>I would use decimal internally in TeX instead of binary</em></p></blockquote> + +<h4 id="question-how-to-record-history">question: how to record history?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>a video &lsquo;Lets not dumb down the history of CS&rsquo; &hellip; used to be history of algorithms &hellip; trouble with funding &hellip; the history is nothing that a non-CS person could not understand &hellip; the whole field of history changed from internal to external &hellip; historians need to be external to get published in journals &hellip; no CS department supports a historian &hellip; recently read a dissertation about the ALGOL 60 copmiler &hellip; very careful, describes data structures and organization &hellip; this kind of thing is what deserves to be called history</em></p></blockquote> + +<h4 id="question-teachers">question: teachers</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>hardest thing for me is choosing between two hypotheses (1) could teach this to anyone (2) only 2% of the world is geeks &hellip; suppose the second is true then you can&rsquo;t talk about how to teach if the teacher is not in the 2% &hellip;</em></p> + <p><em>the newest issue of CACM has a fun typo, &lsquo;deep earning&rsquo;</em></p></blockquote> + +<h3 id="panel-quantum-computing-far-away-around-the-corner-or-maybe-both-at-the-same-time">panel: Quantum Computing: Far Away? Around the Corner? Or Maybe Both at the Same Time?</h3> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>goal to have a 45&ndash;50 qbit machine &hellip; 1 error per 1000 operations &hellip; to test, run sample algorithm, chart output vs. a classical supercomputer &hellip; got to be a supercomputer to finish the computation in time</em></p></blockquote> + +<h5 id="andrew-yao">Andrew Yao</h5> + +<blockquote> + <p><em>I&rsquo;m a believer &hellip; one suggested benchmark is to factor 1000-digit numbers &hellip; impossible to attain &hellip; need to expore new possibilities, take physics attitute</em></p> + <p><em>CS did well attracting attention to quantum &hellip; science should be more open &hellip; share results between physics chemistry CS &hellip; don&rsquo;t just stick to your specialized conferences</em></p> + <p><em>CS departments reception to quantum is less than satisfactory &hellip; 15 years ago, maybe 4 or 5 universities &hellip; now, maybe 7 or 8 .. China doing much better in this regard</em></p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>not useful to make analogy to anything classical &hellip; universal fault tolerance? or computation in the presence of error &hellip; either would be excellent, still a long way off</em></p> + <p><em>IBM put quantum on the cloud &hellip; picked an instruction set that tries to abstract away &hellip; have been 19 published papers on the behavior of this quantum hardware</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>two paths &hellip; finding algorithms, besides Shor&rsquo;s algorithm &hellip; make quantum computer to realize the algorithms &hellip; finding algorithms is very difficult &hellip; information-processing point-of-view</em></p> + <p><em>error correction still small scale &hellip; can we use entanglement between probes to improve accuracy?</em></p></blockquote> + +<h5 id="umesh-vazirani">Umesh Vazirani</h5> + +<blockquote> + <p>_different goals &hellip; maybe you want perfect Qbits for a perfect Hilbert space &hellip; reality is a noisy space &hellip; short run, how to compute with noise &hellip; how to correct errors &hellip;</p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta:</h5> + +<blockquote> + <p>_those 2 paths are the same to me &hellip; we want larger devices with fidelity</p> + <p><em>lets build hardware see where goes &hellip; exciting prospect, computer scientists will explore what they can do with these erroneous qbits &hellip; that&rsquo;s why IBM has the instruction set open to the community</em></p></blockquote> + +<h4 id="question-why-isnt-adding-10-qbits-only-10x-harder">question: why isn&rsquo;t adding 10 qbits only 10x harder?</h4> + +<h5 id="john-martinis">John Martinis:</h5> + +<blockquote> + <p><em>building infrastructure to scale &hellip; not just grad student code &hellip; we&rsquo;re all good coders using standard industry practices for coding</em></p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>fidelity is hard to achieve</em></p></blockquote> + +<h4 id="question-both-ibm-and-google-use-superconducting-storage">question: both IBM and Google use superconducting storage?</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>superconducting scales &hellip; ion traps harder to scale, but we still watch, keep eye on data</em></p></blockquote> + +<h4 id="question-education">question: education</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>I like talking to engineering colleges &hellip; physics and engineering need to work together</em></p></blockquote> + +<h4 id="question-is-quantum-going-to-change-programing-languages">question: is quantum going to change programing languages?</h4> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>yes very different to handle errors &hellip; current challenge is building an abstraction over the superconducting hardware</em></p></blockquote> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>hoping to first expose hardware, then get a model, eventually a language</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>need to start with more algorithms</em></p></blockquote> + +<h4 id="question-what-would-feynman-do">question: what would Feynman do?</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>experiments!</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>yes he&rsquo;d tell us to keep playing, and play with us</em></p></blockquote> + +<h3 id="panel-augmented-reality-from-gaming-to-cognitive-aids-and-beyond">panel: Augmented Reality: From Gaming to Cognitive Aids and Beyond</h3> + +<p>Peter Lee starts off wearing a headset.</p> + +<h5 id="ivan-sutherland">Ivan Sutherland:</h5> + +<blockquote> + <p><em>I can tell you how VR started. Bell Helicopter company wanted to land at night &hellip; put an infrared camera on the landing site and a display in the cockpit &hellip; to test they used the roof of their building &hellip; one day an observer in a Bell office is watching, though the camera, two men playing catch on the roof &hellip; one player threw the ball at the camera and the observer ducked &hellip; he had identified his position with the camera &hellip; my observation was that you didn&rsquo;t need a camera, could substitute a computer &hellip; the rest is history</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>my goal is to augment people &hellip; <a href="https://en.wikipedia.org/wiki/The_Mother_of_All_Demos">Englebart</a> very inspiring &hellip; ok 2 stories &hellip; (1) a student of mine wanted to help picky eaters &hellip; computer vision for when they tried to hide peas under the plate &hellip; projected colors onto the peas, called them &lsquo;disco peas&rsquo;, kids were distracted enough to get over their aversion &hellip; children and parents got involved, new social interaction &hellip; (2) opera makeup for schoolchildren, virtually getting into character &hellip; teenage boys in the classes got to try makeup for the first time &hellip; singers found it useful for rehearsals</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>I feel socially awkward wearing this headset, but I have some of my slides here &hellip; making a wearable headset requires huge effort &hellip; research prototypes can be uncomfortable &hellip; a product needs to be perfect and its very hard to do perfect &hellip; one goal, give Lowe&rsquo;s VR to demo a virtual kitchen &hellip; Case Western anatomy class used virtual cadaver, great collective experience</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>two stories &hellip; (1) Henry Fuchs 1998, working with breast surgeon, try augmented reality to improve the precision of biopsy probe insertion &hellip; 2 years to a working prototype, hard to track surgeon&rsquo;s eyes, display where probe is, where ultrasound is, provide low latency &hellip; one day trying on live patient, worked 100% perfect probe right on the mark, jubilation &hellip; then the doctor had to tell the patient &lsquo;yes it is really cancer&rsquo; &hellip; (2) a challenge, augmented reality EMT training &hellip; real teams, virtual patient, virtual surround &hellip; track real tools, 8 eyes, 8 images, team needs to interact</em></p></blockquote> + +<h4 id="question-what-are-current-uses-of-augmented-reality">question: what are current uses of augmented reality?</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>the pilot of a jumbo jet typically has 1 hour flight experience before he flies for the first time, but extensive training in a flight simulator</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>the <strong>best</strong> AR</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>once I was in a flight simulator with the chief pilot &hellip; and he turned to me and asked &lsquo;have you ever experienced a slow roll in a 747?&rsquo; &hellip; a slow roll is a twisting motion, a very benign maneuver, constant one-G pressure the plane doesn&rsquo;t know its upside down &hellip; &lsquo;here we go&rsquo; and suddenly the world inverted &hellip; I remarked that it was certainly impressive, but didn&rsquo;t you treat the simulator as a real experience, and never attempted anything you would not do in reality? &hellip; &lsquo;that is true, but I am the chief pilot&rsquo;</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>construction, architecture</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>where&rsquo;s the &lsquo;augmented&rsquo;?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>whether augmented or virtual</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>yes we did my kitchen that way, made my wife sick when she tried it</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>surgical</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>still sounds virtual</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>displays on a car, superimposed directions on the tarmac &hellip; one of the users took a vacation and had to use the old GPS technology &hellip; found it very difficult to go back</em></p></blockquote> + +<h4 id="question-ar-tools-for-developers">question: AR tools for developers?</h4> + +<h5 id="blair-macintyre">Blair MacIntyre</h5> + +<blockquote> + <p><em>can developers write apps for the Microsoft <a href="https://www.microsoft.com/en-us/hololens">Hololens</a>?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>we belive in experience, anything we can do to foster experiences is good</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>faking things &hellip; subtle and important &hellip; I remember using a flight simulator, navigating the runway, and I turned my head to see if my wing was going to clip a plane &hellip; turned and there was nothing there &hellip; emotional shock to leave the simulation, I had been flying for 1 hour</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>pilot training is an early adopter because the cost of real planes is so high, impossible to train for emergency situations</em></p> + <p><em>the ultimate goal, you can sit in a virtual chair &hellip; and if the chair has handcuffs you cannot get up &hellip; a virtual bullet is lethal &hellip; probably impossible because bits don&rsquo;t weigh anything &hellip; you know Ben Franklin invented augmented reality, eyeglasses &hellip; the desire outweighs cost &hellip; I cannot see the audience here, maybe it would be good if I had a headset! but Peter took his off</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>because of my slides I couldn&rsquo;t see the audience, but then without the headset I couldn&rsquo;t see them either</em></p></blockquote> + +<h4 id="question-any-challenges-to-suggest-to-the-audience">question: any challenges to suggest to the audience?</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>if we had holographic transport, we wouldn&rsquo;t need planes!</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>maybe, but you need to give attendees a physical presence &hellip; smell, touch</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>what makes us willing to work together? I had a collaboration with three people &hellip; all in different locations .. communicated with a phone &hellip; worked perfectly, because we had worked in the same location first and knew one another so well &hellip; how to get to that point, where a simulation could be a useful tool &hellip; another good observation by Fred Brooks, given a domain X ask how good does the simulation need to be for X &hellip; Licklider told me, you&rsquo;d need damn good fiction to land someone on the moon, the simulation would need to provide every detail &hellip; for flight simulation the user&rsquo;s imagination can fill some gaps, a pilot can recognize an aircraft carrier from a rougher picture</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>at IBM I once hired buses to bring the Poughkeepsie secretaries to the main office &hellip; the secretaries at the two offices only knew one another from the phone &hellip; this one lunch did so much good &hellip; only $75 to rent a bus</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>how important is it to shake hands, to bump into a table?</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>for this conference, I think the live stream is getting a better experience because the cameras zoom in on us, the panelists &hellip; the audience in the back cannot see us, only a picture of us on the monitors</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p>_one excellent video game, starts in the dark, you hear a voice &hellip; turn around and there&rsquo;s a character sitting on a chair &hellip; if you rearrange your furniture he finds a new seat &hellip;</p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>games are a great example &hellip; Pokemon Go &hellip; Apple jusr released an app toolkit &hellip; need to get those in schools, in the hands of kids who can build with them</em></p></blockquote> + +<h4 id="question-ivan-about-your-ultimate-display-paper-what-has-since-surprised-or-frustrated-you">question: Ivan, about your &lsquo;ultimate display&rsquo; paper, what has since surprised or frustrated you?</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>I wasn&rsquo;t surprised because I never had any expectations &hellip; of course sticks are not real &hellip; no assumptions so no strong feelings</em></p></blockquote> + +<h4 id="question-people-already-distracted-by-cell-phones-how-to-manage-all-this-input">question: people already distracted by cell phones, how to manage all this input?</h4> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>good question, how much data you can present to people &hellip; and then the problem with google glass, your companions don&rsquo;t know what you are looking at &hellip; at least with snapchat glasses, you can trust the device is simpler</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>good writing defines reality, bad writing reports it &hellip; with the printing press, quickly went from 30,000 books to over 13,000,000 &hellip; novels evolved shortly after, a new form of expression</em></p></blockquote> + +<h4 id="question-peter-how-long-do-your-people-wear-the-hololens">question: Peter, how long do your people wear the hololens?</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>hard to say &hellip; but often longer than the battery lasts</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>how long does it last?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>depends what you&rsquo;re doing, 3 hours</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>that&rsquo;s encouraging, we had a 30-minute cutoff because participants had enough</em></p></blockquote> + +<h4 id="question-nausea">question: nausea</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>I get nauseous in our minecraft VR &hellip; but there&rsquo;s a pop-out feature where you keep playing, but the game world is in a TV set instead of around you &hellip; can pop back in when you&rsquo;re feeling better</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>we&rsquo;ve seen about 20% of the population gets nauseous</em></p></blockquote> + +<h5 id="audience-member">audience member</h5> + +<blockquote> + <p><em>Dana Boyd conducted an experiment, found the nausea was worse for wemon</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>oculus makes me feel sick, but the hololens has never given me trouble</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>have models to predict head motion, to keep the VR world steadier</em></p></blockquote> + +<h5 id="blair-macintyre">Blair MacIntyre</h5> + +<blockquote> + <p><em>I remember reading papers that measured framerate &hellip; would be interesting to revisit</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>framerate not important, its the latency that gets you &hellip; one colleague of mine, we call her &lsquo;the canary&rsquo; because she&rsquo;s so sensitive, in fact &hellip;</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>talking about nausea is part of the problem, people experience it more &hellip; every time I talk about it in public my co-workers tell me to stop!</em></p> + <p><em>another cool application, there&rsquo;s a hololens app to give blind people a tour of the Redmond office &hellip; you say a building and it takes you there</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>one challenge, the relative brightness of the real and virtual worlds</em></p></blockquote> + +<h4 id="question-any-last-remarks">question: any last remarks</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p>_I hoped from the beginning that AR would be a teaching tool &hellip; I learned that <code>F = MA</code> not from a book but from a large flywheel in the school&rsquo;s basement &hellip; very substantial inertia &hellip; the greatest value for AR would be to show people things in a way that makes the underlying meaning clear &hellip; what color should the hydrogen atoms in a benzene ring be? the color will be fiction, but the quality of learning will depend on that fiction &hellip; challenge for content makers &hellip; what is the haptic experience of feeling bits?</p></blockquote> + +<h3 id="small-group-session">Small Group Session</h3> + +<p>After the last panel, I got to attend a small group session with other students, Dick Karp, and Don Knuth. It doesn&rsquo;t feel right to summarize or quote from the session, but there&rsquo;s one thing I want to write about.</p> + +<p>During the group session, I said something that I now regret. There was a brief silence as the group changed subjects, and someone suggested that we do a round of introductions. I objected, <em>this will take so long</em>, but in fact the introductions were a very good idea.</p> + +<p>Normally, I don&rsquo;t like introductions because they focus on names, backgrounds, and credentials. I don&rsquo;t care about any of these when I&rsquo;m meeting someone! Rather, I prefer to just talk and by-the-way learn about the other person(s). There&rsquo;s an anaology to double-blind reviewing &mdash; the focus should be content and not credentials.</p> + +<p>These introductions were successful for two reasons. First, they gave everyone in the room a turn to speak, and this seemed to help people join the actual discussion sooner. That was strange to me. I always feel a little nervous the first time I speak up in front of a group, but if I really feel like speaking then I can always get over this little barrier. I guess it&rsquo;s not right to assume the nervousness is &ldquo;little&rdquo; for everyone. Second, the introductions format was &ldquo;say your name and a funny fact&rdquo;. This prompt by itself led to some nice conversation topics:</p> + +<ul> + <li>Could a computer program decide whether a statement was funny or not funny?</li> + <li>What kind of humor works in a classroom? In a textbook?</li> + <li>Would this kind of introduction be acceptable in another era or culture, for instance Victorian England?</li></ul> + +<p>&ldquo;Nice&rdquo; in the sense that everyone could contribute, which was a real challenge. Even the question &ldquo;does anyone have a favorite algorithm?&rdquo; didn&rsquo;t have much success fostering discussion.</p> + +<p>Related: a useful greeting at the event was &ldquo;what SIG are you?&rdquo;. The answer was a good hint about what level of abstraction you two could best communicate at.</p> +<!-- ### Misc.--> +<!-- I noticed that some of the young people who served on panels and also gave--> +<!-- long-and-not-very-insightful answers to questions were later on their laptops--> +<!-- as other panels discussed things. I noticed some of the older people who--> +<!-- served on panels falling asleep during other panels--> + + Building a Website with Scribble + http://prl.ccs.neu.edu/blog/2017/05/23/building-a-website-with-scribble/?utm_source=Author-Ben-Greenman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-23-building-a-website-with-scribble + Tue, 23 May 2017 01:53:13 UT + Ben Greenman + +<p>The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!</p> +<!-- more--> + +<h2 id="the-story-so-far">The Story so Far</h2> + +<p>Last Fall, I took a flight to Chicago (on my way to <a href="http://con.racket-lang.org/2016/">RacketCon 2016</a>). When I landed, there was a new message in my inbox:</p> + +<pre><code> Subject: Web Page + Date: 2016-09-15 + + You have been nominated webmaster by public acclamation. Congratulations!</code></pre> + +<p>Emboldened by the trust of my people, I promptly converted the PRL website from Racket-generating-HTML to the fine <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> preprocessor language (commit <a href="https://github.com/nuprl/website/commit/a0600d32fec4bd70c5530b2717aec32979d634f7"><code>a0600d</code></a>) This bold action polarized the community.</p> + +<blockquote> + <p>I can&rsquo;t read the source anymore! Is this really an improvement?</p></blockquote> + +<p>Fear not, citizens. The switch to <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> was the right choice, and you too can learn to read the source code.</p> + +<h2 id="how-to-read-scribblehtml-programs">How to Read <code>scribble/html</code> Programs</h2> + +<h3 id="basics">Basics</h3> + +<p>Scribble is a language for writing Racket documentation. The key innovation in Scribble is the <em>@-expression</em> (read: &ldquo;at expression&rdquo;). The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language combines @-expression syntax with functions that generate HTML.</p> + +<h4 id="-syntax">@-syntax</h4> + +<p><a href="http://www.greghendershott.com/2015/08/at-expressions.html">Greg Hendershott</a> and the <a href="http://docs.racket-lang.org/scribble/reader.html">Scribble Documentation</a> explain @-expressions properly. Here&rsquo;s a short tutorial (Part 1 of 2, &ldquo;the basics&rdquo;):</p> + +<ul> + <li>Scribble programs start in &ldquo;text mode&rdquo;. Every character you type goes straight to the document you are building.</li> + <li>The @-sign toggles to &ldquo;Racket mode&rdquo; for the next expression. In Racket mode, the characters you type will be evaluated as a Racket program to produce part of the document.</li></ul> + +<p><em>Examples:</em> Evaluating <code>"Hello Dave"</code> puts &ldquo;Hello Dave&rdquo; in your document. Evaluating <code>"Hello @Dave"</code> puts &ldquo;Hello ???&rdquo; in your document, where "???" is the value of the variable <code>Dave</code>. Finally if <code>Dave</code> is the name of a function, then <code>"Hello @(Dave)"</code> calls the <code>Dave</code> function with zero arguments and puts whatever it returns into your document.</p> + +<p>To make it easy to interleave text, function calls, and code, Scribble discriminates between 4 kinds of parentheses when they follow an @-sign (Part 2 of 2, &ldquo;the parens&rdquo;):</p> + +<ul> + <li><code>@(f A B)</code> is just like the function call <code>(f A B)</code> in Racket</li> + <li><code>@f[A B]</code> is the same as <code>@(f A B)</code>, but typically more useful because &hellip;</li> + <li><code>@f[A B]{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; to a list of words <code>w*</code>, then calls <code>f</code> just like <code>(apply f A B w*)</code></li> + <li><code>@f{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; and calls <code>f</code> with the results</li> + <li><code>@f|{....}|</code> is similar, but the <code>....</code> are in &ldquo;unescapable text mode&rdquo;</li></ul> + +<p>&ldquo;Unescapable text mode&rdquo; treats @-signs as text instead of toggling between modes.</p> + +<h4 id="generating-html">Generating HTML</h4> + +<p>The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language comes with functions that render HTML. These functions have the same name as the corresponding HTML tag.</p> + +<p>Example program:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this program prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>No surprises.</p> + +<p>One thing that <em>is</em> surprising is how <code>scribble/html</code> handles tag attributes. Every tag-rendering function accepts &ldquo;Racket mode&rdquo; arguments that specify an attribute name and attribute value.</p> + +<p>For example:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">[</span><span class="n">style:</span> <span class="s2">"color:red"</span><span class="p">]{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Hope the output looks familiar. The input syntax is strange, but that&rsquo;s what it is.</p> + +<p>Larger programs print larger webpages. Each page on the PRL website is HTML generated by one <code>scribble/html</code> program.</p> + +<h2 id="why-scribblehtml-is-an-improvement">Why <code>scribble/html</code> is an Improvement</h2> + +<p>Before <code>scribble/html</code>, the PRL website was implemented in <code>scribble/text</code>. A <code>scribble/text</code> program renders and prints text. There is no extra support for HTML.</p> + +<p>To compare, here&rsquo;s the start of the old homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/text</span> +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="s2">"templates.rkt"</span><span class="p">)</span> + +<span class="n">&lt;!DOCTYPE</span> <span class="n">html&gt;</span> +<span class="n">&lt;html</span> <span class="n">lang=</span><span class="s2">"en"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">header</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;body</span> <span class="n">id=</span><span class="s2">"pn-top"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">navbar</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;div</span> <span class="n">class=</span><span class="s2">"jumbotron"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And here is the start of the <code>scribble/html</code>&rsquo;d homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@require</span><span class="p">[</span><span class="s2">"templates.rkt"</span><span class="p">]</span> + +<span class="n">@doctype</span><span class="p">{</span><span class="n">html</span><span class="p">}</span> +<span class="n">@html</span><span class="p">[</span><span class="n">lang:</span> <span class="s2">"en"</span><span class="p">]{</span> + <span class="n">@header</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@body</span><span class="p">[</span><span class="n">id:</span> <span class="s2">"pn-top"</span><span class="p">]{</span> + <span class="n">@navbar</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@div</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"jumbotron"</span><span class="p">]{</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The pages look similar. The new one has more @-signs and parentheses, the old one has more <code>&lt;</code>-signs and quotes. If you were able to edit the old page, you should be able to edit the new page.</p> + +<p>The <strong>key improvement</strong> in the new page is that <strong>common mistakes are now compile-time errors</strong>.</p> + +<ul> + <li> + <p>Before, a typo like <code>&lt;hmtl&gt;</code> would generate an ugly webpage. After, a typo like <code>@hmtl</code> is a syntax error.</p></li> + <li> + <p>Before, a typo like <code>&lt;b&gt;....</code> with no closing tag would generate an ugly webpage. After, a typo like <code>@b{....</code> is a syntax error.</p></li></ul> + +<p>Both flavors of error message come with source-code line numbers. This is very very helpful.</p> + +<h3 id="small-improvements">Small Improvements</h3> + +<h4 id="1-more-functions">1. More Functions</h4> + +<p>Before, the <a href="http://prl.ccs.neu.edu/teaching.html">Teaching page</a> contained some interesting HTML for rendering vertical text (look for the word &ldquo;Semantics&rdquo; to see how this was used):</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"how-to-design-programs"</span><span class="p">&gt;</span>S<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>e<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>m<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>a<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>n<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>t<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>i<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>c<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>s<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>After, the same text is generated from a function call:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@span</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"how-to-design-programs"</span><span class="p">]{</span><span class="n">@vertical-text</span><span class="p">{</span><span class="n">Semantics</span><span class="p">}}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The <code>vertical-text</code> function is simple:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@require</span><span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._only-in))" style="color: inherit">only-in</a></span> <span class="n">racket/list</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span><span class="p">)]</span> + +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">vertical-text</span> <span class="o">.</span> <span class="n">str*</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-~3elist))" style="color: inherit">string-&gt;list</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._append*))" style="color: inherit">append*</a></span> <span class="n">str*</span><span class="p">))</span> <span class="p">(</span><span class="n">br</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="2-more-structure-less-boilerplate">2. More Structure, Less Boilerplate</h4> + +<p>Here&rsquo;s part of the old definition of &ldquo;Ben Greenman&rdquo; on the <a href="http://prl.ccs.neu.edu/people.html">People page</a>:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"row pn-person"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-row-eq-height"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-photo"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"img-wrapper"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">"img/ben_greenman.jpg"</span> <span class="na">title</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="na">alt</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="p">/&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-9"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-4 pn-contact"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"pn-name"</span><span class="p">&gt;</span>Ben Greenman<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + Advisor: Matthias Felleisen<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"mailto:types@"</span><span class="err">@"</span><span class="na">ccs</span><span class="err">.</span><span class="na">neu</span><span class="err">.</span><span class="na">edu</span><span class="err">"</span><span class="p">&gt;</span>types@"@"ccs.neu.edu<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"http://www.ccs.neu.edu/home/types"</span><span class="p">&gt;</span>www.ccs.neu.edu/home/types<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-muted col-md-offset-5"</span><span class="p">&gt;</span> + Joined 2014 + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-bio"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>I like constructions .... <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new definition uses a helper function with keyword arguments for each &ldquo;field&rdquo; of the person:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@person</span><span class="p">[</span><span class="kd">#:name</span> <span class="s2">"Ben Greenman"</span> + <span class="kd">#:title</span> <span class="s2">"Advisor: Matthias Felleisen"</span> + <span class="kd">#:e-mail</span> <span class="s2">"types@ccs.neu.edu"</span> + <span class="kd">#:website</span> <span class="s2">"http://ccs.neu.edu/home/types"</span> + <span class="kd">#:history</span> <span class="n">@list</span><span class="p">[</span><span class="s2">"Joined 2014"</span><span class="p">]</span> + <span class="kd">#:img</span> <span class="s2">"ben_greenman.jpg"</span><span class="p">]{</span> + <span class="n">I</span> <span class="n">like</span> <span class="n">constructions</span> <span class="n">....</span> +<span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="3-less-string-formatting">3. Less String-Formatting</h4> + +<p>Before, the code did a lot of string formatting (<a href="https://github.com/nuprl/website/commit/a0600d#diff-1921e33ce89be28dd277cf1c7880d1beL9">link</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/creatingunits.html#(form._((lib._racket/unit..rkt)._link))" style="color: inherit">link</a></span> <span class="n">url</span> <span class="n">body</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-append))" style="color: inherit">string-append</a></span> <span class="s2">"&lt;a href=</span><span class="se">\"</span><span class="s2">"</span> <span class="n">url</span> <span class="s2">"</span><span class="se">\"</span><span class="s2">&gt;"</span> <span class="n">body</span> <span class="s2">"&lt;/a&gt;"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new code has no need for such helper functions.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@a</span><span class="p">[</span><span class="n">href:</span> <span class="n">url</span> <span class="n">body</span><span class="p">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="bottom-line">Bottom Line</h4> + +<p>Scribble is a good language for making static HTML pages.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://docs.racket-lang.org/pollen/index.html">Pollen</a></li> + <li><a href="https://github.com/vishesh/racketscript">RacketScript</a></li> + <li>Other websites built using <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a>: (1) <a href="http://nanopass.org/">nanopass.github.io</a> (<a href="https://github.com/nanopass/nanopass.github.io">source code</a>), (2) <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> (<a href="https://github.com/nuprl/gtp">source code</a>).</li> + <li><a href="http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/">Notes from a Gradual Typing Across the Spectrum PI meeting</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Daniel-Patterson.atom.xml b/blog/feeds/Author-Daniel-Patterson.atom.xml new file mode 100644 index 00000000..efe88d60 --- /dev/null +++ b/blog/feeds/Author-Daniel-Patterson.atom.xml @@ -0,0 +1,108 @@ + + + PRL Blog: Posts tagged 'Author: Daniel Patterson' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Daniel-Patterson-html + 2018-04-23T10:07:48Z + + [How to prove a compiler fully abstract (cross-post)](https://dbp.io/essays/2018-04-19-how-to-prove-a-compiler-fully-abstract.html) + + urn:http-prl-ccs-neu-edu:-blog-2018-04-23-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html + 2018-04-23T10:07:48Z + 2018-04-23T10:07:48Z + + Daniel Patterson + + + [How to prove a compiler correct (cross-post)](https://dbp.io/essays/2018-01-16-how-to-prove-a-compiler-correct.html) + + urn:http-prl-ccs-neu-edu:-blog-2018-01-17-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html + 2018-01-17T20:58:48Z + 2018-01-17T20:58:48Z + + Daniel Patterson + + + Artifacts for Semantics + + urn:http-prl-ccs-neu-edu:-blog-2017-05-15-artifacts-for-semantics + 2017-05-15T10:08:31Z + 2017-05-15T10:08:31Z + + Daniel Patterson + +<p><a href="http://gallium.inria.fr/~scherer/">Gabriel Scherer</a> and I recently wrote an <a href="https://dbp.io/artifacts/funtal">artifact</a> for a semantics <a href="https://dbp.io/pubs/2017/funtal.pdf">paper</a> on a typed assembly language interoperating with a high-level functional language.</p> +<!-- more--> + +<p>We wrote a interpreter, typechecker, and parser in OCaml, compiled it to Javascript using <a href="http://ocsigen.org/js_of_ocaml/">js_of_ocaml</a>, and then put it on a webpage (with an editor with syntax highlighting and error reporting) that allows people to step through examples from the paper or write their own. (Feel free to start by playing a bit with <a href="https://dbp.io/artifacts/funtal">our artifact</a>).</p> + +<p>This post will summarize the different parts to make it easier for others to repeat this process. We think it was a total success, and have gotten feedback that it makes understanding the (somewhat complex) language from the paper much easier. We argue that building such interpreters / typecheckers is easy enough that all papers should do this. Further, while our interpreter / typechecker is completely unverified, since we wrote it in OCaml, this approach should work equally well for semantics verified in Coq and then extracted to OCaml.</p> + +<hr /> + +<p>The paper in question, <a href="https://dbp.io/pubs/2017/funtal.pdf">FunTAL: Reasonably Mixing a Functional Language with Assembly</a> (to appear in PLDI17), presents a multi-language that incorporates a typed assembly language (TAL) and a simple functional language where each can be embedded within the other. The paper then develops a logical relation that can be used to reason about the equivalence of such mixed programs. For example in the paper we show an imperative register-based factorial and a functional factorial equivalent.</p> + +<p>Both the static and dynamic semantics are relatively complex. The typed assembly has registers (which store word-sized values), a heap (which stores code-blocks and tuples), and a stack (not a call-stack, simply a place where word-sized values can be pushed and popped). Code-blocks have pre-conditions on the state of the registers and the stack, and allow the tail of the stack to be abstracted over polymorphically. This allows values to be protected on the stack before jumping to blocks that otherwise could change them. This is used, along with a novel notion of <strong>return markers</strong>, to ensure well-bracketing in the control flow of the typed assembly. The return markers indicate the location that points to the block that will eventually be returned to (assuming it doesn&rsquo;t loop infinitely). At the top level, the return marker <code>end</code> indicates that, assuming it does not loop, eventually the program will stop, rather than returning somewhere else.</p> + +<p>Understanding the dynamic semantics requires tracking how values flow through the registers, the heap, and the stack, and rather than a call-stack, the user has to track the control flow through the statically-enforced return markers. This allows a good deal of low-level control-flow while still ensuring that calls will eventually return to the right place. This well-bracketing is vital to be able to reason about &ldquo;components&rdquo; that eventually return a value of a particular type, a necessity when embedding these components in a typed high-level program! However, it does mean that understanding the static and dynamic semantics from a few rules alone is a tall order. Our functional language is more standard, though we use (iso)-recursive types to allow recursion, which can easily trip up people, especially when you don&rsquo;t have a type-checker to catch typos!</p> + +<p>For that reason, when working through examples for the paper I implemented a simple interpreter for the multi-language. I did this in OCaml, in the most straightforward way possible: by translating the definitions from the paper into type definitions (<a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/ftal.ml#L835">for F</a> and <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/ftal.ml#L1209">for TAL</a>), and the reduction relation into a <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/ftal.ml#L1155">&ldquo;step&rdquo; function</a> that (assuming it wasn&rsquo;t stuck or a value), did one step of evaluation. Later, I did the same thing for the type-checker, translating rules into a <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/ftal.ml#L282">type-checking function</a>. The latter had to deviate from the rules in the paper in a few minor ways, as the typing rules we had in the paper were slightly not syntax directed.</p> + +<p>Having the interpreter and type-checker was very useful for me, as I could check that the examples from the paper did not contain typos, but it was much less useful as an artifact for a reader of the paper. To use it the reader would have to download the source, install OCaml, write out examples as OCaml data constructors in a test file, compile it, run it, and then interpret the (quite overwhelming) output of every step of evaluation. At each step, I printed the current registers, current stack, current heap, what the evaluation context was (as you might be evaluating TAL instructions that were embedded inside a functional program that, in turn, was embedded in further TAL instructions), and what the current reduction was.</p> + +<p>To get from that useful-for-the-authors artifact to a useful-to-readers artifact requires doing three things:</p> + +<ol> + <li>Allow reading/writing programs in a notation as close to the paper as possible. In our paper we use superscripts, subscripts, and a few greek letters, but ended up with a syntax otherwise very close to the paper &mdash; the biggest differences were a few extra delimiters introduced to reduce ambiguity.</li> + <li>Present an interface that highlights type errors at the location they occurred in, and allow a reader to step forward and backwards through the evaluation. Printing console output traces is fine for authors, but adds too much effort for readers.</li> + <li>Put it online! Don&rsquo;t require installing any software! Conveniently, implementing 2 is also made easier once done online, as we could use existing editor tooling to present the code, highlight errors, etc. By using OCaml, we were able to easily use the excellent <a href="http://ocsigen.org/js_of_ocaml/">js_of_ocaml</a>.</li></ol> + +<p>The first was done by Gabriel, who wrote a grammar using <a href="http://gallium.inria.fr/~fpottier/menhir/">Menhir</a>, and then equipped it with custom parsing error messages that provide much better feedback when there are typos in what people are trying. We also wrote a pretty-printer using the <a href="http://gallium.inria.fr/blog/first-release-of-pprint/">PPrint</a> library, so we could show intermediate program states through the UI. After writing this, we were able to convert our existing suite of test cases and examples to be written textually, which was a huge improvement for us as well! These and other tests were used to ensure that the parser/pretty-printer would round-trip properly.</p> + +<p>For the interface, I built a simple web page that had the <a href="https://codemirror.net/">CodeMirror</a> editor equipped with a very simple syntax highlighter (<a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/artifact/index.html#L247">8 lines of code</a> to highlight keywords &amp; atoms, plus a CodeMirror extension to highlight matching brackets) and error highlighting (which is triggered by the OCaml code). I then made a simple &ldquo;machine state&rdquo; UI that showed, in pretty-printed format, the heap, stack, registers, context, and redex. On the OCaml side, when the &ldquo;run&rdquo; button is clicked, we parse and typecheck and, assuming no errors occur, store the current state as our &ldquo;history&rdquo;. As the user clicks forward or backwards, we run the step function and append to the history of states or pop states off of the history. In total, there are <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/artifact/index.html#L246">50 lines of Javascript</a> and about <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/web.ml">150 lines of OCaml</a> that handle the logic for this interactive UI.</p> + +<p>Putting it online was very easy, based on the choice of tools used earlier. We compile the main file (<a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/web.ml">web.ml</a>) to Javascript using <a href="http://ocsigen.org/js_of_ocaml/">js_of_ocaml</a>, and it pulls in the parser, type-checker, interpreter, examples, etc. The rest of the artifact is a single html file, a CSS file, and a few javascript files for CodeMirror. It requires no server backend, is easy to archive and save, and will even run on smartphones!</p> + +<p>The total time spent implementing the artifact was a small fraction of the time spent on the paper (probably 15 days of person-time), and while it was not in any critical way essential for the success of the paper, it does make the paper much easier to read, and we would argue that all semantics papers would be better off with easy to use artifacts for experimentation. Also, while implementing the artifact we found a few mistakes in the typing judgments for the language. The most interesting one was for our <code>protect</code> TAL instruction, which exists to protect the tail of the stack in a fresh type variable. We had written this as a typing rule that type-checked the rest of the instruction sequence with the abstracted tail, but this never allowed the tail to be accessed again. By translating the typing judgments exactly into code, we realized that there was a problem, because examples that should have worked did not type-check! We were then able to fix the typing rule to conform to what we originally thought it achieved &mdash; locally abstracting, but not abstracting from outside the component. What is interesting is that this did not come up in our proofs because the typing rule was perfectly valid &mdash; it just did not allow non-trivial programs that used the <code>protect</code> instruction. It&rsquo;s quite possible we would have noticed this without implementing the artifact, but the artifact certainly made it easier!</p> + +<p>To see the artifact online, visit:</p> + +<p><a href="https://dbp.io/artifacts/funtal">https://dbp.io/artifacts/funtal</a></p> + +<p>The source code is at:</p> + +<p><a href="https://github.com/dbp/funtal/tree/032be70f33f77e80f4fab7e62016bfabf96476f3">https://github.com/dbp/funtal</a></p> + + Linear Types for Low-level Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-linear-types-for-low-level-languages + 2017-02-28T09:51:55Z + 2017-02-28T09:51:55Z + + Daniel Patterson + <!-- more--> + +<p>In this talk, we covered early papers (primarily, by Girard, Lafont, and Abramsky) on linear logic and its reflections into computation. The goal was to understand why linearity is often turned to as a principled way to control resource usage, as shows up in a language like Rust. From the very beginning, researchers realized the implications for &ldquo;low-level&rdquo; languages - that linear resources would eliminate the need for garbage collection, allow in-place mutation, and enable safe parallel computation. However, pure implementations of linearity incur lots of copying, doing away with any efficiency gained, and we covered a survey of papers that attempted to reconcile this contradiction by weakening linearity in controlled ways.</p> + +<p>Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-02-14.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;02&ndash;14.md</a></li></ul> + +<hr /> + +<p>Just after the talk, over lunch, we had a lab discussion about the phrase &ldquo;low level&rdquo;. Here are some thoughts:</p> + +<ul> + <li>the phrase is relative, both over time and depending on the programming task at hand</li> + <li>a &ldquo;low level&rdquo; task is &ldquo;one that you shouldn&rsquo;t need to worry about&rdquo; while solving your current task</li></ul> + +<p>And here are some example &ldquo;low-level&rdquo; tasks:</p> + +<ul> + <li>Time and space management is &ldquo;low level&rdquo; when designing a new algorithm (the first question is correctness)</li> + <li>Calling conventions and endian-ness (facets of the damn machine running the programs) are almost always low-level</li> + <li>Whether a given value is serializable is usually low-level</li> + <li>Possible side effects, thrown exceptions, and optional arguments can all be considered &ldquo;low level&rdquo; aspects of library functions. This is low-level in the sense that &ldquo;I&rsquo;d rather use a simpler type to think about this library&rdquo;</li> + <li>Managing type annotations is a low-level detail in ML programs</li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Daniel-Patterson.rss.xml b/blog/feeds/Author-Daniel-Patterson.rss.xml new file mode 100644 index 00000000..ba397d72 --- /dev/null +++ b/blog/feeds/Author-Daniel-Patterson.rss.xml @@ -0,0 +1,102 @@ + + + + PRL Blog: Posts tagged 'Author: Daniel Patterson' + PRL Blog: Posts tagged 'Author: Daniel Patterson' + http://prl.ccs.neu.edu/blog/tags/Author-Daniel-Patterson.html + Mon, 23 Apr 2018 10:07:48 UT + Mon, 23 Apr 2018 10:07:48 UT + 1800 + + [How to prove a compiler fully abstract (cross-post)](https://dbp.io/essays/2018-04-19-how-to-prove-a-compiler-fully-abstract.html) + http://prl.ccs.neu.edu/blog/2018/04/23/-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html/?utm_source=Author-Daniel-Patterson&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-04-23-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html + Mon, 23 Apr 2018 10:07:48 UT + Daniel Patterson + + + [How to prove a compiler correct (cross-post)](https://dbp.io/essays/2018-01-16-how-to-prove-a-compiler-correct.html) + http://prl.ccs.neu.edu/blog/2018/01/17/-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html/?utm_source=Author-Daniel-Patterson&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-01-17-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html + Wed, 17 Jan 2018 20:58:48 UT + Daniel Patterson + + + Artifacts for Semantics + http://prl.ccs.neu.edu/blog/2017/05/15/artifacts-for-semantics/?utm_source=Author-Daniel-Patterson&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-15-artifacts-for-semantics + Mon, 15 May 2017 10:08:31 UT + Daniel Patterson + +<p><a href="http://gallium.inria.fr/~scherer/">Gabriel Scherer</a> and I recently wrote an <a href="https://dbp.io/artifacts/funtal">artifact</a> for a semantics <a href="https://dbp.io/pubs/2017/funtal.pdf">paper</a> on a typed assembly language interoperating with a high-level functional language.</p> +<!-- more--> + +<p>We wrote a interpreter, typechecker, and parser in OCaml, compiled it to Javascript using <a href="http://ocsigen.org/js_of_ocaml/">js_of_ocaml</a>, and then put it on a webpage (with an editor with syntax highlighting and error reporting) that allows people to step through examples from the paper or write their own. (Feel free to start by playing a bit with <a href="https://dbp.io/artifacts/funtal">our artifact</a>).</p> + +<p>This post will summarize the different parts to make it easier for others to repeat this process. We think it was a total success, and have gotten feedback that it makes understanding the (somewhat complex) language from the paper much easier. We argue that building such interpreters / typecheckers is easy enough that all papers should do this. Further, while our interpreter / typechecker is completely unverified, since we wrote it in OCaml, this approach should work equally well for semantics verified in Coq and then extracted to OCaml.</p> + +<hr /> + +<p>The paper in question, <a href="https://dbp.io/pubs/2017/funtal.pdf">FunTAL: Reasonably Mixing a Functional Language with Assembly</a> (to appear in PLDI17), presents a multi-language that incorporates a typed assembly language (TAL) and a simple functional language where each can be embedded within the other. The paper then develops a logical relation that can be used to reason about the equivalence of such mixed programs. For example in the paper we show an imperative register-based factorial and a functional factorial equivalent.</p> + +<p>Both the static and dynamic semantics are relatively complex. The typed assembly has registers (which store word-sized values), a heap (which stores code-blocks and tuples), and a stack (not a call-stack, simply a place where word-sized values can be pushed and popped). Code-blocks have pre-conditions on the state of the registers and the stack, and allow the tail of the stack to be abstracted over polymorphically. This allows values to be protected on the stack before jumping to blocks that otherwise could change them. This is used, along with a novel notion of <strong>return markers</strong>, to ensure well-bracketing in the control flow of the typed assembly. The return markers indicate the location that points to the block that will eventually be returned to (assuming it doesn&rsquo;t loop infinitely). At the top level, the return marker <code>end</code> indicates that, assuming it does not loop, eventually the program will stop, rather than returning somewhere else.</p> + +<p>Understanding the dynamic semantics requires tracking how values flow through the registers, the heap, and the stack, and rather than a call-stack, the user has to track the control flow through the statically-enforced return markers. This allows a good deal of low-level control-flow while still ensuring that calls will eventually return to the right place. This well-bracketing is vital to be able to reason about &ldquo;components&rdquo; that eventually return a value of a particular type, a necessity when embedding these components in a typed high-level program! However, it does mean that understanding the static and dynamic semantics from a few rules alone is a tall order. Our functional language is more standard, though we use (iso)-recursive types to allow recursion, which can easily trip up people, especially when you don&rsquo;t have a type-checker to catch typos!</p> + +<p>For that reason, when working through examples for the paper I implemented a simple interpreter for the multi-language. I did this in OCaml, in the most straightforward way possible: by translating the definitions from the paper into type definitions (<a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/ftal.ml#L835">for F</a> and <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/ftal.ml#L1209">for TAL</a>), and the reduction relation into a <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/ftal.ml#L1155">&ldquo;step&rdquo; function</a> that (assuming it wasn&rsquo;t stuck or a value), did one step of evaluation. Later, I did the same thing for the type-checker, translating rules into a <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/ftal.ml#L282">type-checking function</a>. The latter had to deviate from the rules in the paper in a few minor ways, as the typing rules we had in the paper were slightly not syntax directed.</p> + +<p>Having the interpreter and type-checker was very useful for me, as I could check that the examples from the paper did not contain typos, but it was much less useful as an artifact for a reader of the paper. To use it the reader would have to download the source, install OCaml, write out examples as OCaml data constructors in a test file, compile it, run it, and then interpret the (quite overwhelming) output of every step of evaluation. At each step, I printed the current registers, current stack, current heap, what the evaluation context was (as you might be evaluating TAL instructions that were embedded inside a functional program that, in turn, was embedded in further TAL instructions), and what the current reduction was.</p> + +<p>To get from that useful-for-the-authors artifact to a useful-to-readers artifact requires doing three things:</p> + +<ol> + <li>Allow reading/writing programs in a notation as close to the paper as possible. In our paper we use superscripts, subscripts, and a few greek letters, but ended up with a syntax otherwise very close to the paper &mdash; the biggest differences were a few extra delimiters introduced to reduce ambiguity.</li> + <li>Present an interface that highlights type errors at the location they occurred in, and allow a reader to step forward and backwards through the evaluation. Printing console output traces is fine for authors, but adds too much effort for readers.</li> + <li>Put it online! Don&rsquo;t require installing any software! Conveniently, implementing 2 is also made easier once done online, as we could use existing editor tooling to present the code, highlight errors, etc. By using OCaml, we were able to easily use the excellent <a href="http://ocsigen.org/js_of_ocaml/">js_of_ocaml</a>.</li></ol> + +<p>The first was done by Gabriel, who wrote a grammar using <a href="http://gallium.inria.fr/~fpottier/menhir/">Menhir</a>, and then equipped it with custom parsing error messages that provide much better feedback when there are typos in what people are trying. We also wrote a pretty-printer using the <a href="http://gallium.inria.fr/blog/first-release-of-pprint/">PPrint</a> library, so we could show intermediate program states through the UI. After writing this, we were able to convert our existing suite of test cases and examples to be written textually, which was a huge improvement for us as well! These and other tests were used to ensure that the parser/pretty-printer would round-trip properly.</p> + +<p>For the interface, I built a simple web page that had the <a href="https://codemirror.net/">CodeMirror</a> editor equipped with a very simple syntax highlighter (<a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/artifact/index.html#L247">8 lines of code</a> to highlight keywords &amp; atoms, plus a CodeMirror extension to highlight matching brackets) and error highlighting (which is triggered by the OCaml code). I then made a simple &ldquo;machine state&rdquo; UI that showed, in pretty-printed format, the heap, stack, registers, context, and redex. On the OCaml side, when the &ldquo;run&rdquo; button is clicked, we parse and typecheck and, assuming no errors occur, store the current state as our &ldquo;history&rdquo;. As the user clicks forward or backwards, we run the step function and append to the history of states or pop states off of the history. In total, there are <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/artifact/index.html#L246">50 lines of Javascript</a> and about <a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/web.ml">150 lines of OCaml</a> that handle the logic for this interactive UI.</p> + +<p>Putting it online was very easy, based on the choice of tools used earlier. We compile the main file (<a href="https://github.com/dbp/funtal/blob/032be70f33f77e80f4fab7e62016bfabf96476f3/web.ml">web.ml</a>) to Javascript using <a href="http://ocsigen.org/js_of_ocaml/">js_of_ocaml</a>, and it pulls in the parser, type-checker, interpreter, examples, etc. The rest of the artifact is a single html file, a CSS file, and a few javascript files for CodeMirror. It requires no server backend, is easy to archive and save, and will even run on smartphones!</p> + +<p>The total time spent implementing the artifact was a small fraction of the time spent on the paper (probably 15 days of person-time), and while it was not in any critical way essential for the success of the paper, it does make the paper much easier to read, and we would argue that all semantics papers would be better off with easy to use artifacts for experimentation. Also, while implementing the artifact we found a few mistakes in the typing judgments for the language. The most interesting one was for our <code>protect</code> TAL instruction, which exists to protect the tail of the stack in a fresh type variable. We had written this as a typing rule that type-checked the rest of the instruction sequence with the abstracted tail, but this never allowed the tail to be accessed again. By translating the typing judgments exactly into code, we realized that there was a problem, because examples that should have worked did not type-check! We were then able to fix the typing rule to conform to what we originally thought it achieved &mdash; locally abstracting, but not abstracting from outside the component. What is interesting is that this did not come up in our proofs because the typing rule was perfectly valid &mdash; it just did not allow non-trivial programs that used the <code>protect</code> instruction. It&rsquo;s quite possible we would have noticed this without implementing the artifact, but the artifact certainly made it easier!</p> + +<p>To see the artifact online, visit:</p> + +<p><a href="https://dbp.io/artifacts/funtal">https://dbp.io/artifacts/funtal</a></p> + +<p>The source code is at:</p> + +<p><a href="https://github.com/dbp/funtal/tree/032be70f33f77e80f4fab7e62016bfabf96476f3">https://github.com/dbp/funtal</a></p> + + Linear Types for Low-level Languages + http://prl.ccs.neu.edu/blog/2017/02/28/linear-types-for-low-level-languages/?utm_source=Author-Daniel-Patterson&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-linear-types-for-low-level-languages + Tue, 28 Feb 2017 09:51:55 UT + Daniel Patterson + <!-- more--> + +<p>In this talk, we covered early papers (primarily, by Girard, Lafont, and Abramsky) on linear logic and its reflections into computation. The goal was to understand why linearity is often turned to as a principled way to control resource usage, as shows up in a language like Rust. From the very beginning, researchers realized the implications for &ldquo;low-level&rdquo; languages - that linear resources would eliminate the need for garbage collection, allow in-place mutation, and enable safe parallel computation. However, pure implementations of linearity incur lots of copying, doing away with any efficiency gained, and we covered a survey of papers that attempted to reconcile this contradiction by weakening linearity in controlled ways.</p> + +<p>Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-02-14.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;02&ndash;14.md</a></li></ul> + +<hr /> + +<p>Just after the talk, over lunch, we had a lab discussion about the phrase &ldquo;low level&rdquo;. Here are some thoughts:</p> + +<ul> + <li>the phrase is relative, both over time and depending on the programming task at hand</li> + <li>a &ldquo;low level&rdquo; task is &ldquo;one that you shouldn&rsquo;t need to worry about&rdquo; while solving your current task</li></ul> + +<p>And here are some example &ldquo;low-level&rdquo; tasks:</p> + +<ul> + <li>Time and space management is &ldquo;low level&rdquo; when designing a new algorithm (the first question is correctness)</li> + <li>Calling conventions and endian-ness (facets of the damn machine running the programs) are almost always low-level</li> + <li>Whether a given value is serializable is usually low-level</li> + <li>Possible side effects, thrown exceptions, and optional arguments can all be considered &ldquo;low level&rdquo; aspects of library functions. This is low-level in the sense that &ldquo;I&rsquo;d rather use a simpler type to think about this library&rdquo;</li> + <li>Managing type annotations is a low-level detail in ML programs</li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Dustin-Jamner.atom.xml b/blog/feeds/Author-Dustin-Jamner.atom.xml new file mode 100644 index 00000000..03611980 --- /dev/null +++ b/blog/feeds/Author-Dustin-Jamner.atom.xml @@ -0,0 +1,84 @@ + + + PRL Blog: Posts tagged 'Author: Dustin Jamner' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Dustin-Jamner-html + 2017-06-16T11:38:25Z + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Dustin-Jamner.rss.xml b/blog/feeds/Author-Dustin-Jamner.rss.xml new file mode 100644 index 00000000..507ac7a2 --- /dev/null +++ b/blog/feeds/Author-Dustin-Jamner.rss.xml @@ -0,0 +1,76 @@ + + + + PRL Blog: Posts tagged 'Author: Dustin Jamner' + PRL Blog: Posts tagged 'Author: Dustin Jamner' + http://prl.ccs.neu.edu/blog/tags/Author-Dustin-Jamner.html + Fri, 16 Jun 2017 11:38:25 UT + Fri, 16 Jun 2017 11:38:25 UT + 1800 + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=Author-Dustin-Jamner&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Gabriel-Scherer.atom.xml b/blog/feeds/Author-Gabriel-Scherer.atom.xml new file mode 100644 index 00000000..95e02a4d --- /dev/null +++ b/blog/feeds/Author-Gabriel-Scherer.atom.xml @@ -0,0 +1,1233 @@ + + + PRL Blog: Posts tagged 'Author: Gabriel Scherer' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Gabriel-Scherer-html + 2017-08-13T14:29:41Z + + Reviews and author responses: we should stop asking for 500-word responses + + urn:http-prl-ccs-neu-edu:-blog-2017-08-13-reviews-and-author-responses-we-should-stop-asking-for-500-word-responses + 2017-08-13T14:29:41Z + 2017-08-13T14:29:41Z + + Gabriel Scherer + +<p>This year I reviewed many ICFP submissions, and got to be on the receiving end of equally many author responses (also sometimes called, somewhat combatively, rebuttals). I found that there was a large difference between the official written advice on author responses and what I, as a reviewer reading the responses, found effective. In particular, I now believe that limiting yourself to 500 words should strongly be avoided &mdash; we should even stop giving that advice.</p> +<!-- more--> + +<p>This year, I had the honor (and accompanying load of work) of being a Program Committee (PC) member at the ICFP conference. It was my first time being a PC member at a conference, and I found it extremely pleasant and interesting, thanks to the authors who sent us articles to review, my fellow PC members, and the ever-smiling careful balancing work of our PC chair, Mark Jones. It was also a lot of work, starting with 18 reviews to do over a month and a half, an intense PC meeting, and the new &ldquo;second phase&rdquo; process with the opportunity for authors and reviewers to exchange feedback on changes requested by the program committee.</p> + +<p>There is little guidance on how to write author responses, although this <a href="http://www.pl-enthusiast.net/2014/09/17/advice-writing-author-response/">blog post</a> by Michael Hicks on pl-enthusiast is quite good. One thing that is obvious as a reviewer and is only slightly brushed in this post, however, is that author responses should <em>not</em> aim to fit a 500 words limit, and in fact I believe that it is a bad idea to do so.</p> + +<p>As for most conference, the ICFP review system recommends (in writing) to authors to keep their response to 500 words (some systems also highlight words after those in red to make the point clear). Don&rsquo;t do this! The least convincing responses I have seen are those that followed this recommendation.</p> + +<p>(I have also seen at least 18*2 other reviewers read the same responses I read, most of them well over 500 words, and <em>none</em> of them made any comment on the length of the author responses.)</p> + +<p>We have a frustrating situation where the explicit rule is different from the thing people do in practice. This is bad for newcomers that do not know the norms and cannot tell if ignoring the rule may hurt them. This is the point of this blog post:</p> + +<ul> + <li> + <p>If you are an author, please know that disrespecting the 500-words limit is the <em>right thing</em> to do. You should also know, of course, that people have limited time, so keep the main body of your response reasonably long. (I spent a day and a half on your paper already, I am willing to spend 10 additional minutes reading the response.)</p></li> + <li> + <p>If you are a program committee chair or a Magical HotCRP Wizard, please remove this silly recommendation to keep responses to 500 words. (See an alternative proposal at the end of this post.)</p></li></ul> + +<h2 id="my-personal-response-format">My personal response format</h2> + +<p>My author responses start with general comments that elaborate on the main point I want to tell all reviewers. Then, a second part contains per-reviewer comments (one section per reviewer); it is clearly marked as skippable. Here is the skeleton of the last response I wrote:</p> + +<blockquote> + <p>We thank the reviewers for their work on our article and their detailed feedback. We start with a general discussion that responds to the salient points raised by reviewers. In a second part, we provide detailed responses to the questions/remarks of each reviewer.</p> + <h3 id="general-discussion">General discussion</h3> + <p>[..]</p> + <h3 id="specific-questionscomments-review-a">Specific questions/comments: review #A</h3> + <p>[..]</p> + <h3 id="specific-questionscomments-review-b">Specific questions/comments: review #B</h3> + <p>[..]</p> + <h3 id="specific-questionscomments-review-c">Specific questions/comments: review #C</h3> + <p>[&hellip;]</p></blockquote> + +<p>For this particular response, the &ldquo;General discussion&rdquo; section used 1296 words according to <code>wc -w</code> (M-x shell-command-on-region). In the following sections, I quote the reviews to answer specific points, email-style (following the Markdown syntax that HotCRP renders properly).</p> + +<h2 id="suggested-wording-for-pc-chairs">Suggested wording for PC chairs</h2> + +<p>If you are a PC chair, you should remove the suggestion of respecting a 500 words limit for your conference. Here would be a suggested alternative wording:</p> + +<blockquote> + <p>Please remember, in writing your author response, that reviewers may stop reading the response at any point. We suggest having a reasonably-sized main section where you make your most important high-level comments, and clearly marked sections where you answer individual reviewer&rsquo;s questions. It is not useful nor productive to answer every point of each review, you should focus on the comments that you believe the reviewers are most interested in.</p></blockquote> + + Syntactic parametricity strikes again + + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-syntactic-parametricity-strikes-again + 2017-06-05T14:27:44Z + 2017-06-05T14:27:44Z + Gabriel Scherer + Li-Yao Xia + + Gabriel Scherer, Li-Yao Xia + +<p>In this blog post, reporting on a collaboration with <a href="https://poisson.chat/">Li-Yao Xia</a>, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely &ldquo;syntactic&rdquo; way, by looking at the structure of canonical derivations. More precisely, I prove that \( +\newcommand{\List}[1]{\mathsf{List}~#1} +\newcommand{\Fin}[1]{\mathsf{Fin}~#1} +\newcommand{\Nat}[1]{\mathbb{N}} +\newcommand{\rule}[2]{\frac{\displaystyle \array{#1}}{\displaystyle #2}} +\newcommand{\judge}[2]{{#1} \vdash {#2}} +\newcommand{\emptyrule}[1]{\begin{array}{c}\\[-1em] #1 \end{array}} + ∀α. \List α → \List \alpha +\) is isomorphic to \( + Π(n:\Nat{}). \List{(\Fin{n})} +\) where \(\Fin{n}\) is the type of integers smaller than \(n\), corresponding to the set \(\{0, 1, \dots, n-1\}\).</p> +<!-- more--> + +<p>Context: Last week I had the pleasure of visiting UPenn, where I had many interesting discussions with various people. It was also an occasion to temporarily resume a discussion/collaboration I have with Li-Yao Xia, who is currently an intern there, and Jean-Philippe Bernardy, about testing polymorphic programs and its relation to canonical representations for System F.</p> + +<p>During one of our blackboard discussion, Li-Yao and I did a manual proof of a cool result: we proved a parametricity theorem for \(∀α. \List α → \List α\) using syntactic methods, namely proof search among canonical proofs. (This is an idea that I have been playing with since the last year of my <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/">PhD thesis</a>, where I unsuccessfully tried to extend my work on canonical forms for the simply-typed lambda-calculus to polymorphism. It is here worked out on an specific example, but my end goal is to turn the manual reasoning into an algorithm.)</p> + +<p>You may wonder, first, why the isomorphism holds. The idea is that a polymorphic function of type \(\List α → \List α\) cannot inspect the elements of the input list; it can only use them in the resulting list, possibly duplicating, reordering or dropping some elements. On any input list of size \(n\), the behavior of the function can be described by a list of indices in \([0; n-1]\). For example, if the input \([x, y, z]\) (for some values of \(x, y, z\)) gives the output \([y, y, x]\), then this relation will hold on <em>any</em> value of \(x, y, z\), as the function cannot inspect their value or even test them for equality. The behavior of this function on lists of size 3 can be fully described by the list of indices \([1, 1, 0]\). Its whole behavior is then uniquely determined by one such list for each possible size:</p> + +<p>\[ + ∀α. \List α → \List α \quad≃\quad Π(n:\Nat{}). \List{(\Fin n)} +\]</p> + +<p>The idea behind the &ldquo;syntactic&rdquo; (proof-theoretic?) proof method is the following: the set of closed values at a type \(A\) is isomorphic to the <em>search space</em> for canonical/normal derivations of \(\judge{}{A}\). We have tools (in particular the notion of <em>invertible</em> inference rules) to reason on those – in this post I will only present the reasoning informally, but it can easily be made formally precise.</p> + +<p>We start by looking at the shape of the search space for</p> + +<p>\[ + \judge{}{∀α. \List α → \List α} +\] or, said otherwise, of the judgment \[ + \judge{}{\List α → \List α} +\]</p> + +<p>with a fresh/abstract type variable \(α\). (I will not be keeping opened type variables in context to avoid confusing them with hypotheses.)</p> + +<p>Any derivation of a function type, without loss of generality (w.l.o.g), is equivalent to a derivation starting with a function introduction. This is the η-expansion rule for functions: any proof term \(e\) is equivalent to \(λx.~(e~x)\), a proof that starts with a \(λ\). So any proof can be taken to start as follows: \[ +\rule{ +\judge{\List \alpha}{\List \alpha} +}{ +\judge{}{\List \alpha \to \List \alpha} +} +\] we can, w.l.o.g, unfold the recursive type in the context (\(\List α = 1 + (α × \List α)\)): \[ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}} +\]</p> + +<p>A derivation with a sum type as hypothesis can, w.l.o.g, be assumed to start by splitting on this pair (this is the η-expansion rule for sums): \[ +\rule{ +\judge{1}{\List α} +\quad +\judge{α × \List α}{\List α} +}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]</p> + +<p>In the right subgoal, we can always, w.l.o.g, split a hypothesis of product type: \[ +\rule{ +\emptyrule{\judge{1}{\List α}} +\quad +\rule{ +\judge{α, \List α}{\List α} +}{ +\judge{α × \List α}{\List α} +}}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]</p> + +<p>Now, an interesting pattern emerges. In the process of trying to prove \(\judge{\List α}{\List α}\), we have to prove the (right) subgoal \(\judge{α,\List α}{α}\). We can generalize this derivation by assuming that we start with some number \(n\) of variables of type \(α\) in the context (we write \(α^n\) for this): \[ +\rule{ +\rule{ +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n, 1}{\List \alpha} +} +\quad +\rule{ +\judge{\alpha^{n+1}, \List \alpha}{\List \alpha} +}{ +\judge{\alpha^n, \alpha \times \List \alpha}{\List \alpha} +}}{ +\rule{ +\judge{\alpha^n, 1 + (\alpha \times \List \alpha)}{\List \alpha} +}{ +\judge{\alpha^n, \List \alpha}{\List \alpha} +}} +\]</p> + +<p>\[ +\newcommand{\llbracket}{〚} +\newcommand{\rrbracket}{〛} +\newcommand{\sem}[1]{\llbracket{} #1 \rrbracket{}} +\]</p> + +<p>Let us write \(\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}}\) for the search space corresponding to all possible derivations of the judgment \(\judge{\alpha^n, \List \alpha}{\List \alpha}\). All the proof steps above have been done &ldquo;without loss of generality&rdquo; (in terms of focusing, we only used invertible rules), so they appear in any such derivation. Similarly, let us write \(\sem{\judge{\alpha^n}{\List \alpha}}\) for the space of all possible derivations of \(\judge{\alpha^n}{\List \alpha}\), then above we have proven that \[ +\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} +\quad=\quad +\sem{\judge{\alpha^n}{\List \alpha}} +\times +\sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} +\]</p> + +<p>This equality can be unfolded at will \[ +\begin{align} +&amp; \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+3}, \List \alpha}{\List \alpha}} \\ += &amp; \dots \\ +\end{align} +\]</p> + +<p>or written as an infinite product \[ + \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} + \quad=\quad + \prod_{k \in \Nat{}}{\sem{\judge{\alpha^{n+k}}{\List \alpha}}} +\] and, in particular, \[ +\begin{align} +&amp; \sem{\judge{}{\List \alpha \to \List \alpha}} \\ += &amp; \sem{\judge{\alpha^0, \List \alpha}{\List \alpha}} \\ += &amp; \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ +\end{align} +\]</p> + +<p>Now let&rsquo;s look at the structure of the derivations of \(\judge{\alpha^n}{\List \alpha}\). A proof of this judgment cannot start with a &ldquo;left rule&rdquo;, inspecting the value of one of the \(n\) variables of type \(α\), given that the structure of \(α\) is unknown/abstract. It must start by choosing to either build the empty list or a cons cell. We write this as follows (after unfolding the type):</p> + +<p>\[ +\rule{ +\rule{ +\judge{\alpha^n}{1} +\quad\oplus\quad +\judge{\alpha^n}{\alpha \times \List \alpha} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]</p> + +<p>The \(\oplus\) notation between two judgments is non-standard; it means that they are not two requirements of the same proof, but two alternatives for possible proofs. All valid proofs fit that structure, and they either have a \(\judge{\alpha^n}{1}\) premise or a \(\judge{\alpha^n}{\alpha \times \List \alpha}\) premise. With this syntax, we are describing a set of possible derivations, rather than a single (partial) derivation.</p> + +<p>Proofs of \(\judge{\Gamma}{1}\) are trivial, and a proof of a product is always, w.l.o.g, a product of proofs (in intuitionistic logic / the λ-calculus they reuse the same context), so we can decompose further: \[ +\rule{ +\rule{ +\rule{ +}{ +\judge{\alpha^n}{1} +} +\quad\oplus\quad +\rule +{ +\judge{\alpha^n}{\alpha} +\quad +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n}{\alpha \times \List \alpha} +} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]</p> + +<p>There is exactly one possible proof of \(\judge{\alpha^n}{1}\), so its search space is \(1\), the unit set (with a single element). There are exactly \(n\) possible proofs of \(\judge{\alpha^n}{\alpha}\), so the search space is just \(n\), seen as a set, or, in type-theoretic notation, \(\Fin{n}\). We thus have the recursive equation: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +1 + (\Fin n \times \sem{\judge{\alpha^n}{\List \alpha}}) +\]</p> + +<p>This type is either \(1\), or a \(\Fin{n}\) and itself, recursively. This is exactly a list: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +\List{(\Fin{n})} +\]</p> + +<p>so, plugging everything together: \[ +\begin{align} +&amp; \sem{\forall \alpha. \List \alpha \to \List \alpha} \\ += &amp; \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ += &amp; \prod_{n \in \Nat{}}{\List{(\Fin{n})}} \\ +\end{align} +\]</p> + +<h3 id="post-scriptum">Post Scriptum</h3> + +<p>Some of reasoning steps above can be formulated in a way that is less clear but more familiar, as a sequence of type isomorphisms. For example, the first part on \(\sem{\judge{\alpha^n, \List +\alpha}{\List \alpha}}\) can written as:</p> + +<p>\[ +\begin{align} +&amp; +∀α. αⁿ × \List α → \List α +\\ &amp; += \text{(unfold List)} +\\ &amp; + ∀α. αⁿ × (1 + α × \List α) → \List α +\\ &amp; + = \text{(distribute × over +)} +\\ &amp; + ∀α. ((αⁿ × 1) + (αⁿ⁺¹ × \List α)) → \List α +\\ &amp; + = \text{(A × 1 ≃ A)} +\\ &amp; + ∀α. (αⁿ + (αⁿ⁺¹ × \List α)) → \List α +\\ &amp; + = \text{(A+B) → C ≃ (A→C)×(B→C)} +\\ &amp; + ∀α. (αⁿ → \List α) × (αⁿ⁺¹ × \List α → \List α) +\\ &amp; + = \text{(distribute ∀α below product)} +\\ &amp; + (∀α. αⁿ → \List α) × (∀α. αⁿ⁺¹ × \List α → \List α) +\\ +\end{align} +\]</p> + +<p>Reading this equational sequence, it may look like we had to make the right choice at each step; but the proof-search perspective reveals that there were in fact no choices, as each time we apply invertible rules (&ldquo;w.l.o.g. rules&rdquo;).</p> + +<p>Furthermore, some parts cannot be derived in this style; in the latter part of the proof, the isomorphism between \(∀\alpha. \alpha^n → \alpha\) and \(\Fin{n}\), which is immediate from a proof search perspective, cannot be justified in this way. (In particular, \(A×B → C\) is <em>not</em> isomorphic to \((A→C)+(B→C)\).)</p> + +<h3 id="going-further">Going further</h3> + +<ul> + <li> + <p>It is an unfortunately-little-known obvious fact that many things we associate to &ldquo;free theorems&rdquo; can be recovered by proof search. For example, it is much simpler to prove that the only inhabitant of \(\forall \alpha. \alpha \to \alpha\) is the identity using proof search than parametricity. I briefly discussed the idea in the section 1.3 of my 2015 article, <a href="http://gallium.inria.fr/~scherer/research/unique_inhabitants/unique_stlc_sums-long.pdf">Which simple types have a unique inhabitant?</a>.</p></li> + <li> + <p>If you are unfamiliar with proof search (or the LF community) and curious about what I mean by &ldquo;canonical forms&rdquo; and why I think this is an important idea, see my non-technical 2017 article <a href="http://www.ccs.neu.edu/home/gasche/research/canonical-forms/snapl.pdf">Search for Program Structure</a>. The problem of extending the notion of canonical forms to arbitrary polymorphic types is briefly discussed in the section 14.5 of my 2016 <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">phd manuscript</a>.</p></li> + <li> + <p>If you haven&rsquo;t heard of it yet, you would probably be interested in the 2010 article <a href="http://publications.lib.chalmers.se/records/fulltext/local_99387.pdf">Testing Polymorphic Properties</a> by Jean-Philippe Bernardy, Patrik Jansson and Koen Claessen. Li-Yao has a 2016 implementation called <a href="https://github.com/Lysxia/metamorph">Metamorph</a> that got us talking together. The problems of understanding canonical forms and testing are quite related, but yet not exactly the same&hellip;</p></li></ul> + +<h3 id="you-might-also-like">You might also like</h3> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="https://williamjbowman.com/blog/2017/01/03/toward-type-preserving-compilation-of-coq-at-popl17-src/">Toward Type-Preserving Compilation of Coq, at POPL17 SRC</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> + + PRL at SNAPL'17 + + urn:http-prl-ccs-neu-edu:-blog-2017-04-25-prl-at-snapl-17 + 2017-04-25T16:46:54Z + 2017-04-25T16:46:54Z + + Gabriel Scherer + +<p>PRL recently produced three papers for the <a href="http://snapl.org/2017/index.html">SNAPL</a> conference.</p> + +<ul> + <li><a href="https://dbp.io/pubs/2017/linking-types-snapl.pdf">Linking Types for Multi-Language Software: Have Your Cake and Eat It Too</a>, by Daniel Patterson and Amal Ahmed.</li> + <li><a href="http://www.ccs.neu.edu/home/gasche/research/canonical-forms/snapl.pdf">Search for Program Structure</a>, by Gabriel Scherer</li> + <li><a href="http://www.ccs.neu.edu/racket/pubs/typed-racket.pdf">Migratory Typing: Ten Years Later</a>, by Sam Tobin-Hochstadt, Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Ben Greenman, Andrew M. Kent, Vincent St-Amour, T. Stephen Strickland and Asumu Takikawa</li></ul> +<!-- more--> + +<h3 id="httpsdbpiopubs2017linking-types-snaplpdflinking-types-for-multi-language-software-have-your-cake-and-eat--it-too"><a href="https://dbp.io/pubs/2017/linking-types-snapl.pdf">Linking Types for Multi-Language Software: Have Your Cake and Eat It Too</a></h3> + +<p>Daniel Patterson and Amal Ahmed, 2017</p> + +<blockquote> + <p>Software developers compose systems from components written in many different languages. A business-logic component may be written in Java or OCaml, a resource-intensive component in C or Rust, and a high-assurance component in Coq. In this multi-language world, program execution sends values from one linguistic context to another. This boundary-crossing exposes values to contexts with unforeseen behavior—that is, behavior that could not arise in the source language of the value. For example, a Rust function may end up being applied in an ML context that violates the memory usage policy enforced by Rust’s type system. This leads to the question of how developers ought to reason about code in such a multi-language world where behavior inexpressible in one language is easily realized in another.</p> + <p>This paper proposes the novel idea of linking types to address the problem of reasoning about single-language components in a multi-lingual setting. Specifically, linking types allow programmers to annotate where in a program they can link with components inexpressible in their unadulterated language. This enables developers to reason about (behavioral) equality using only their own language and the annotations, even though their code may be linked with code written in a language with more expressive power.</p></blockquote> + +<h3 id="httpwwwccsneueduhomegascheresearchcanonical-formssnaplpdfsearch-for-program-structure"><a href="http://www.ccs.neu.edu/home/gasche/research/canonical-forms/snapl.pdf">Search for Program Structure</a></h3> + +<p>Gabriel Scherer, 2017.</p> + +<blockquote> + <p>The community of programming language research loves the Curry-Howard correspondence between proofs and programs. Cut-elimination as computation, theorems for free, &lsquo;call/cc&rsquo; as excluded middle, dependently typed languages as proof assistants, etc.</p> + <p>Yet we have, for all these years, missed an obvious observation: &ldquo;the structure of <em>programs</em> corresponds to the structure of proof <em>search</em>&rdquo;. For pure programs and intuitionistic logic, more is known about the latter than the former. We think we know what programs are, but logicians know better!</p> + <p>To motivate the study of proof search for program structure, we retrace recent research on applying the logical technique of focusing to study the canonical structure of simply-typed λ-terms. We then motivate the open problem of extending canonical forms to support richer type systems, such as polymorphism, by discussing a few enticing applications of more canonical program representations.</p></blockquote> + +<h3 id="httpwwwccsneueduracketpubstyped-racketpdfmigratory-typing-ten-years-later"><a href="http://www.ccs.neu.edu/racket/pubs/typed-racket.pdf">Migratory Typing: Ten Years Later</a></h3> + +<p>Sam Tobin-Hochstadt, Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Ben Greenman, Andrew M. Kent, Vincent St-Amour, T. Stephen Strickland and Asumu Takikawa, 2017.</p> + +<blockquote> + <p>In this day and age, many developers work on large, untyped code repositories. Even if they are the creators of the code, they notice that they have to figure out the equivalent of method signatures every time they work on old code. This step is time consuming and error prone.</p> + <p>Ten years ago, the two lead authors outlined a linguistic solution to this problem. Specifically they proposed the creation of typed twins for untyped programming languages so that developers could migrate scripts from the untyped world to a typed one in an incremental manner. Their programmatic paper also spelled out three guiding design principles concerning the acceptance of grown idioms, the soundness of mixed-typed programs, and the units of migration.</p> + <p>This paper revisits this idea of a migratory type system as implemented for Racket. It explains how the design principles have been used to produce the Typed Racket twin and presents an assessment of the project’s status, highlighting successes and failures.</p></blockquote> + +<p>.</p> + +<p>SNAPL is not dissimilar to the (french-speaking) <a href="http://jfla.inria.fr/">JFLA</a> that I am more familiar with &mdash; with an added irritating call for paper and unreasonable registration price. It has an interesting diversity of topics of presentation: see also the complete <a href="http://snapl.org/2017/papers.html">list of accepted papers</a> this year, and the <a href="http://snapl.org/2015/papers.html">list of the previous edition</a>.</p> + + Bullets are good for your Coq proofs + + urn:http-prl-ccs-neu-edu:-blog-2017-02-21-bullets-are-good-for-your-coq-proofs + 2017-02-21T19:04:28Z + 2017-02-21T19:04:28Z + + Gabriel Scherer + +<p>I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.</p> + +<p>Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.</p> +<!-- more--> + +<h2 id="why-bullets">Why bullets</h2> + +<p>While you are doing a proof, Coq shows a list of subgoals that have to be proved before the whole proof is complete. Most proof steps will operate on the current active subgoal, changing the hypotheses or the goal to prove, but some proof steps will split it into several subgoals (growing the total list of goals), or may terminate the proof of the current subgoal and show you the next active subgoal.</p> + +<p>Before bullets, a typical proof script would contain the proofs of each subgoal, one after another.</p> + +<pre><code>induction foo. (* this creates many subgoal *) + +proof of first subgoal. + +proof of second subgoal.</code></pre> + +<p>There are many ways to structure this to make the structure more apparent: people would typically have a comment on each subgoal, or make disciplined use of indentation and blank lines. But, in my experience, a major problem with this style was maintainability in the face of changes to the definitions or parts of automation. It could be very hard of what was happening when a proof suddenly broke after a change before in the file:</p> + +<ul> + <li> + <p>If a proof step now proves <em>less</em> things, then what used to be the end of a subgoal may not be anymore. Coq would then start reading the proof of the next subgoal and try to apply it to the unfinished previous goals, generating very confusing errors (you believe you are in the second subgoal, but the context talks about a leaf case of the first goal).</p></li> + <li> + <p>If a proof step now proves <em>more</em> things, it is also very bad! The next proof steps, meant for the first subgoal (for example), would then apply to the beginning of the second subgoal, and you get very confusing errors again.</p></li></ul> + +<p>What we need for robustness is a way to indicate our <em>intent</em> to Coq, when we think that a subgoal is finished and that a new subgoal starts, so that Coq can fail loudly at the moment where it notices that this intent does not match reality, instead of at an arbitrary later time.</p> + +<p>(The <code>S*Case</code> tactics used in (older versions of) Software Foundations can solve this problem if used in a carefully, systematic way, and additionally provides naming. Alexandre Pilkiewicz implemented an even more powerful <a href="https://github.com/pilki/cases">cases</a> plugin. Bullets are available in standard Coq since 8.4 (released in 2012), and can be used with no effort.)</p> + +<p>There is not much discussion of bullets around; see the <a href="https://coq.inria.fr/distrib/8.6/refman/Reference-Manual009.html#sec326">documentation</a> in the Coq manual. I learned a lot from Arthur Azevedo de Amorim&rsquo;s <a href="https://github.com/arthuraa/poleiro/blob/master/theories/Bullets.v">Bullets.v</a> file.</p> + +<p>Finally, some people don&rsquo;t use bullets, because they systematically use so much automation that they never see subgoals &mdash; each lemma has a one-line proof. This is also a valid style. (I have been going to Adam Chlipala&rsquo;s <a href="https://frap.csail.mit.edu/main">Formal Reasoning about Programs</a> 2017 class, where Adam ignores bullets because that is his usual style.) Because I am not crushy enough to do this from the start, my proofs tend to start with cases and subgoals, and then I refine them to add more automation for robustness. I found bullets very useful for the first step, and during the refinement process.</p> + +<h2 id="bullets">Bullets</h2> + +<p>Bullets are actually a combination of two features, braces <code>{ ... }</code> and actual list bullets &mdash; <code>-</code>, <code>+</code>, <code>*</code>, or homogeneous repetitions of those, for example <code>--</code> or <code>***</code>.</p> + +<h3 id="braces">Braces</h3> + +<p>The opening brace <code>{</code> focuses the proof on the current subgoal. If you finish the proof of the subgoal, the following subgoal will not become accessible automatically; you have to use the closing brace <code>}</code> first. (If you finish the goal earlier than you think, you get an error.) Conversely, <code>}</code> fails if the subgoal is not complete. (If you fail to finish, you get an error.)</p> + +<p>The previous example can thus be written as follows, and will be more robust:</p> + +<pre><code>induction foo. (* this creates many subgoal *) +{ + proof of first subgoal. +} +{ + proof of second subgoal. +}</code></pre> + +<p>If you also want to make sure that an error occurs if the number of subgoals changes (for example if new constructors are added to the inductive type of <code>foo</code>), you can use an outer layer of braces:</p> + +<pre><code>{ induction foo. (* this creates many subgoal *) + { + proof of first subgoal. + } + { + proof of second subgoal. + } +} (* would fail if a new subgoal appeared *)</code></pre> + +<h3 id="list-bullets">List bullets</h3> + +<p>A bullet, for example <code>--</code>, also focuses on the next subgoal. The difference is that when the subgoal is finished, you do not have a closing construction, you must use the same bullet to move to the next subgoal. (Again, this fails if the first proof step changes to prove too much or too little.) With bullets you would write</p> + +<pre><code>induction foo. (* this creates many subgoal *) ++ proof of first subgoal. ++ proof of second subgoal.</code></pre> + +<p>Bullets can be nested, but you must use different bullets for the different nesting levels. For example, if this proof is only one subgoal of a larger proof, you can use:</p> + +<pre><code>- induction foo. (* this creates many subgoal *) + + proof of first subgoal. + + proof of second subgoal. +- (* would fail if a new subgoal appeared *) + rest of the proof</code></pre> + +<p>The natural ordering of bullets, I think, is by increasing number of lines: <code>-</code>, <code>+</code> then <code>*</code> (and then multi-character bullets, I guess). You can also mix bullets with braces: the opening brace resets the bullet scope, any bullet can be used again with the subgoal.</p> + +<p>This gives a large space of freedom in how you want to use these features. You can use only braces, only bullets, braces and only one level of bullets, etc. My own style evolved with experience using the feature, and I will present the current status below.</p> + +<h2 id="my-current-bullet-style">My current bullet style</h2> + +<p>When deciding how to use bullets, one distinguishes the commands that preserve the number of subgoals and those that may create new subgoals. I use some additional distinctions.</p> + +<p>Some tactics, for example <code>assert</code>, create a number of subgoals that is <em>statically</em> known, always the same for the tactic. I then use braces around each sub-proof, except the last one, which I think of as the &ldquo;rest&rdquo; of the current proof.</p> + +<pre><code>assert foo as H. +{ proof of foo. } +rest of the proof using H:foo.</code></pre> + +<p>(If the proof of <code>foo</code> takes several lines, I two-indent them, with the braces alone on their lines.)</p> + +<p>Most tactics create a <em>dynamic</em> number of subgoals, that depends on the specifics of the objects being operated on; this is the case of <code>case</code>, <code>destruct</code>, <code>induction</code> for example. In this case, I open a brace before the tactic, and use a bullet for each subgoal.</p> + +<pre><code>{ induction foo; simpl; auto. +- proof of first remaining subgoal. +- proof of second remaining subgoal. + rest of the proof of the second subgoal. +}</code></pre> + +<p>(Notice that the subgoal-creating step is vertically aligned with the proof steps: I use both braces and bullets, but take only one indentation level each time.)</p> + +<p>As an exception, I may omit the braces if we are at the toplevel of the proof (<code>Proof .. Qed</code> serve as braces).</p> + +<p>Note that omitting the braces here and using different bullets when you nest is also just fine. In my experience it gives proofs that are a bit more pleasant to read but also a bit more cumbersome to edit and move around.</p> + +<p>Finally, a not-uncommon mode of use of &ldquo;dynamic&rdquo; tactics in the sense above is to expect all the cases, except one, to be discharged by direct automation (for example they are all absurd except one). When it is my intent that all cases but one be discharged (and not a coincidence), I express it by not using braces (this command preserves the number of subgoals), but marking the remaining subgoal with a new bullet <em>without</em> increasing the indentation level.</p> + +<pre><code>{ induction foo. +- first subgoal. +- second subgoal. + case blah; discharge all sub-subgoals but one. ++ remaining sub-subgoal of the second subgoal. + finish the sub-subgoal. +- third subgoal. +}</code></pre> + +<p>(This is the only time where I use several bullet levels.)</p> + +<p>If you are the kind of programmer that is passionate about indentation style, I should now have tricked you to use bullets to propose a different variant. Otherwise, please consider using bullets anyway, for example by following the style above, it will make your life easier in the face of changing developments.</p> + + Measuring the submission/review balance + + urn:http-prl-ccs-neu-edu:-blog-2016-12-17-measuring-the-submission-review-balance + 2016-12-17T16:33:10Z + 2016-12-17T16:33:10Z + + Gabriel Scherer + +<p>How do researchers know whether they are doing &ldquo;enough&rdquo; or &ldquo;too many&rdquo; reviews? A measurable goal is to be review-neutral: to have demanded, through our submissions, as many reviews as we have produced as reviewers.</p> +<!-- more--> + +<h3 id="reviewing-is-good">Reviewing is good</h3> + +<p>I like to review academic papers. It is a very rewarding activity in many different ways. One gets to serve the academic community, helping it function smoothly. One gets a chance at acquiring a much better understanding of someone else&rsquo;s work than idle paper-skimming allows. One gets to send feedback to our colleagues and help them improve their work and its presentation &mdash; it is also an essential way in which we can participate to the formation of student researchers all over the world. Finally, doing reviews helped me develop the skill the judge someone else&rsquo;s work and of forcing oneself to come up with a decisive opinion &mdash; it is surprisingly difficult and only comes with training.</p> + +<p>Doing reviews is also fairly time-consuming. I noticed that the time I spend on each review is generally stable (excursions into previous or related work excluded): around one day and a half for conference reviews, and at least twice more for journal reviews &mdash; I&rsquo;m sure other people have wildly different figures, but I would expect it to be a noticeable time commitment in any case. (Workshop reviews are much easier, at least for the formats I have seen of 2-page extended abstracts, I&rsquo;d say one hour per review.)</p> + +<h3 id="how-many-reviews">How many reviews?</h3> + +<p>Because it is so time-consuming, deciding whether to say &ldquo;yes&rdquo; or &ldquo;no&rdquo; to invitations to review a new paper is not easy: in general I want to say &ldquo;yes&rdquo; (unless I can tell that I will not enjoy reading the paper at all), but it is not reasonable to say &ldquo;yes&rdquo; all the time, because I also need to spend time on other things. When should I say &ldquo;no&rdquo; because I have done &ldquo;too many&rdquo; reviews already?</p> + +<p>We can count the number of reviews that we have done, and we can also estimate the number of reviews that we have demanded of others through our submissions. A natural goal for researchers is to produce at least as many reviews as they demand; if everyone reached this goal, the peer-review system would be at equilibrium without imposing too much of a workload on anyone.</p> + +<p>To estimate the number of reviews a researcher demanded from their peers, you can sum, for each of their submissions to a peer-reviewed venue, the number of reviews that they received, divided by the total number of authors of the submissions.</p> + +<p>\[ \sum_{p \in \mathtt{Submissions}} \frac{\mathtt{reviews}(p)}{\mathtt{authors}(p)} \]</p> + +<p>Out of curiosity, I just measured this balance for myself: over my years doing research I have &ldquo;demanded&rdquo; 10 workshop reviews and 28.5 conference reviews, and &ldquo;produced&rdquo; 6 workshop reviews and 17 conference reviews. If you think that an article would interest me, you shouldn&rsquo;t feel bad about asking me to review it, for now. (On the other hand, my balance <em>this year</em> is positive, so I wouldn&rsquo;t feel to bad about refusing if I had to.)</p> + +<p>Of course, a researcher&rsquo;s balance is highly dependent on where they are in their academic career &mdash; maybe more so that on their personal choices. Students are supposed to submit articles, but are offered few opportunities for doing reviews. When they are invited to do reviews, it is often as sub-reviewer, one review at a time. More established researchers participate in program committees, where they have to do a fair amount of reviews at once &mdash; ten to twenty can be typical in Programming Languages conferences. This means that one naturally starts with a deficit of reviews, and that the opportunity to become balanced or positive only comes over the years.</p> + +<p>(There is much more that could be said about the dynamics of the submission/review balance. I think the idea that a single person should be neutral should not be taken too seriously, because the dynamics are so complex. For example, some people stop doing reviews with a negative balance (students going to the industry for example), so long-time researchers necessarily have a <em>very positive</em> balance that may make short-time researchers balance considerations mostly irrelevant. Another thing is that there is no point doing more reviews than required by the submission flow, and that doing more reviews would build up more reviewing debt under this neutrality criterion &mdash; you can never have everyone positive.)</p> + +<h3 id="quality">Quality</h3> + +<p>This is only a comment on the quantitative aspects of reviewing. Much more important is the qualitative part: are the reviews you receive and produce good reviews? (There is no objective definition of what a good review is; I like reviews that are constructive, help improve the work and its presentation, and catch mistakes.) For a given paper, one or a few very good reviews is more helpful than many bad reviews, so one should not compromise on the quality of one&rsquo;s reviews in order to reach a quantitative goal.</p> + +<h3 id="advice-for-students">Advice for students?</h3> + +<p>While proof-reading this post (thanks!), Ben asked some questions that may be of interest to others &mdash; mostly students, I suppose.</p> + +<blockquote> + <p>If I want to be review-neutral, but I have to accumulate a &ldquo;review debt&rdquo; before I can start reviewing, does this mean I should accept my first opportunity to review and every one that follows (until I&rsquo;m neutral)?</p></blockquote> + +<p>The answer is of course &ldquo;no&rdquo;: one should never feel forced to accept reviews. On the other hand, I do think that it is worthwhile for PhD students to take advantage of the reviews they are offered, so &ldquo;saying yes most of the time&rdquo; sounds like a reasonable strategy to me &mdash; this is just a personal opinion. Some reasons:</p> + +<ul> + <li> + <p>Reviewing is hard and takes training, I think it is good to start practicing early. Students are in a good situation to exercise their reviewing skills at a fairly calm peace (you won&rsquo;t get many reviews anyway), and with more time than more senior people.</p></li> + <li> + <p>Student reviews are often done as sub-reviewer: someone does a review, but also asks for your opinion and includes your sub-review in their review. It is a low-pressure way to do your first reviews, and the ability to exchange opinions with the other reviewer and discuss both reviews is really helpful. Students can also ask for feedback on their reviews to their advisor, which is also very helpful.</p></li> + <li> + <p>Reviewing teaches a few useful things about writing papers as well &mdash; it&rsquo;s always easier to recognize the flaws in others&rsquo; work.</p></li></ul> + +<p>On the other hand, I think you should not accept reviews at times when you cannot invest enough work in the review, or when doing so would be detrimental to you &mdash; whether you are on a deadline, or under too much pressure, or have conflicting commitments, etc. This is more important than anything about a submission/review balance.</p> + +<blockquote> + <p>Do you have any ideas for how young researchers / new researchers can reduce their &ldquo;review footprint&rdquo;? For example, is it possible to volunteer for reviews?</p></blockquote> + +<p>Yes, you can volunteer for reviews by telling the colleagues in your lab that you would be interested in doing reviews and that they should consider giving you some.</p> + +<p>(With the increased use of double-blind submission processes, it is becoming more difficult to pass conference reviews to external researchers. This means that students are relatively unlikely to receive review offers from outside their close colleagues.)</p> + +<p>Besides doing more reviews, the two other factors one could in theory play with are: submitting less papers, and having more co-authors. I think there is something to be said for the first one: one reason to not submit unfinished, buggy or topically-inappropriate articles is that it has a review cost. The second factor should not be considered, I think: &ldquo;did this person contribute to the work?&rdquo; should weight infinitely more for co-authorship decisions.</p> + +<p>Note: Another thing you can ask for is <em>reading reviews other people received</em>. I think that reading reviews is also very helpful for research beginners &mdash; whether reviews of one&rsquo;s own work or someone else&rsquo;s. In particular, I wouldn&rsquo;t know how to write reviews if I hadn&rsquo;t had the opportunity to read reviews before that. If someone you are close to receives reviews, you should consider asking them whether you could have a look.</p> + +<blockquote> + <p>Is being a student volunteer at a conference equal to &ldquo;one review&rdquo;?</p></blockquote> + +<p>I think it is a distinct form of academic service. I don&rsquo;t know how to measure the &ldquo;conference organization cost&rdquo; we impose to our academic colleagues. (If there are around 500 attendants to a typical Programming Languages conference, it means that for every 500 conferences you attend you should organize one all by yourself.)</p> + + SRC-submissions + + urn:http-prl-ccs-neu-edu:-blog-2016-11-17-src-submissions + 2016-11-17T13:52:52Z + 2016-11-17T13:52:52Z + + Gabriel Scherer + +<p>Max New, Daniel Patterson and Ben Greenman recently wrote three two-page abstracts on what they are working on right now. Come have a look &mdash; and any feedback is welcome!</p> +<!-- more--> + +<h2 id="gradual-type-precision-as-retraction">Gradual Type Precision as Retraction</h2> + +<p><a href="http://maxsnew.github.io/docs/precision-as-retraction.pdf">Gradual Type Precision as Retraction</a> + <br />Max New + <br />2016</p> + +<blockquote> + <p>Gradually typed programming languages allow for a mix of precision of static type information, allowing advanced type features to be added to existing languages, while still supporting interoperability with legacy code. The advantages of gradual typing are enticing to researchers and practitioners alike, but a general theory of gradually typed languages is only beginning to emerge after a decade of research.</p> + <p>It has long been noted that there is much similarity between work on contracts and gradual typing, and the use of retracts in domain theory which were used to relate models of untyped and typed lambda calculus in <a href="https://pdfs.semanticscholar.org/359e/ca57fe42d97cbb67f0b5591869abe5eb5421.pdf">Scott(1976)</a> and <a href="http://andrewkish-name.s3.amazonaws.com/scott80.pdf">Scott(1980)</a>. Here we take this connection seriously and consider how judgments in modern gradually typed languages can be framed in terms of retractions. While retractions in programming languages were originally studied in terms of denotational semantics in domains, our presentation will use only the most basic elements of category theory: composition, identity and equality of terms, so our formulation is equally applicable to axiomatic or operational semantics.</p> + <p>In particular we propose a semantic criterion for the notion of precision of gradual types, a common judgment in gradually typed languages (sometimes called naive subtyping for historical reasons). We relate it to a previous definition from <a href="https://www.eecs.northwestern.edu/%7Erobby/pubs/papers/esop2009-wf.pdf">Wadler and Findler(2009)</a> that defines type precision in terms of blame. We show that our definition decomposes in a similar way into “positive” and “negative” type precision, but without depending on a specific notion of blame in the language.</p></blockquote> + +<h2 id="linking-types-specifying-safe-interoperability-and-equivalences">Linking Types: Specifying Safe Interoperability and Equivalences</h2> + +<p><a href="https://dbp.io/pubs/2016/linking-types-poplsrc2017-proposal.pdf">Linking Types: Specifying Safe Interoperability and Equivalences</a> + <br />Daniel Patterson + <br />2016</p> + +<blockquote> + <p>All programs written in high-level languages link with libraries written in lower-level languages, often to expose constructs, like threads, random numbers, or automatic serialization, that aren’t possible in the high-level language. This linking usually takes place after compiling both languages to a common language, possibly assembly. In this sense, reasoning about crosslanguage linking means reasoning about compilation.</p> + <p>While most languages include cross-language linking (FFI) mechanisms, they are ad-hoc and can easily break the semantic equivalences of the source language, making it hard for source programmers to reason about correctness of their programs and hard for compiler writers to reason about correctness of their optimizations.</p> + <p>In this work, I design and motivate linking types, a language-based mechanism for formally specifying safe linking with libraries utilizing features inexpressible in the source. Linking types allows programmers to reason about their programs in the presence of behavior inexpressible in their language, without dealing with the intricacies of either the compiler or the particular language they are linking with.</p></blockquote> + +<h2 id="pruning-contracts-with-rosette">Pruning Contracts with Rosette</h2> + +<p><a href="http://www.ccs.neu.edu/home/types/resources/popl2017-src.pdf">Pruning Contracts with Rosette</a> + <br />Ben Greenman + <br />2016</p> + +<blockquote> + <p><a href="http://www.ccs.neu.edu/racket/pubs/icfp16-dnff.pdf">Contracts</a> are a pragmatic tool for managing software systems, but programs using contracts suffer runtime overhead. If this overhead becomes a performance bottleneck, programmers must manually edit or remove their contracts. This is no good. Rather, the contracts should identify their own inefficiencies and remove unnecessary dynamic checks. Implementing contracts with <a href="https://emina.github.io/rosette/">Rosette</a> is a promising way to build such self-aware contracts.</p></blockquote> + +<h2 id="while-were-at-it-lets-rant-on-srcs">While we&rsquo;re at it let&rsquo;s rant on SRCs</h2> + +<p>These abstracts are submitted at POPL&rsquo;s &ldquo;Student Research Competition&rdquo;. You submit an abstract, and if you get accepted to that thing, you get a bit of travel support money, and you have to prepare a poster and present it at the conference.</p> + +<p>I have a firm dislike for the <em>Competition</em> part of that concept: I think that people think of research too competitively already, and that we should have less of that, not more. (Having some is unfortunately unavoidable in scarce-resource situations.) I think that the process of awarding prizes to students with the &ldquo;best poster&rdquo; is dumb &mdash; and borderline ridiculous.</p> + +<p>On the other hand, my experience seeing them writing these extended abstracts is that it&rsquo;s a useful exercise for them, and produces nice result &mdash; short, readable introductions to their ideas. And Jennifer Paykin <a href="https://github.com/gasche/icfp2016-blog/blob/master/SVs/jennifer_paykin.md">convincingly argues</a> that although writing a poster is rather painful, actually presenting it during the conference is interesting and useful. In her words, &ldquo;it&rsquo;s worth it to get the experience of authentic and fruitful discussions&rdquo;. Plus having posters in the corridors of one&rsquo;s lab is very nice.</p> + +<p>I think we could have &ldquo;Student Research Sessions&rdquo; or &ldquo;Student Poster Sessions&rdquo;, where students are encouraged to present their work, would write those nice extended abstracts and posters, interact with researchers at the conference, and get travel money, without the ranking and prize stuff. (I would still encourage students to participate to SRC today, it seems to be worth it.)</p> + + Emacs daemon for fast editor startup + + urn:http-prl-ccs-neu-edu:-blog-2016-10-17-emacs-daemon-for-fast-editor-startup + 2016-10-17T21:48:25Z + 2016-10-17T21:48:25Z + + Gabriel Scherer + +<p>In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs&rsquo; bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors &mdash; on each git commit for example, or when I use a <a href="https://addons.mozilla.org/en-US/firefox/addon/its-all-text/">Firefox extension</a> to edit a textarea content in a proper editor.</p> + +<p>In this blog post, I describe how to use <code>emacsclient</code> to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.</p> +<!-- more--> + +<p>Emacs has long supported a client/server mode: a daemon emacs instance is loaded in the background, and whenever you request a new emacs window you can creates a new frame (~ window) using the same instance. This means that the startup time is dramatically reduced after the daemon is launched, as for example the execution of your personal configuration code does not have to be repeated.</p> + +<p>To use it, I have this code as <code>/usr/bin/editor</code>:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="ch">#!/bin/bash</span> +emacsclient -a <span class="s2">""</span> -c <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The empty <code>-a</code> parameter means that if no daemon exists, it should start one in the background and retry. The <code>-c</code> option means that a new frame (window) should be created instead of reusing an existing one. <code>"$@"</code>means that when the script is invoked with a path as command-line parameter (<code>editor /tmp/foo.txt</code>), the corresponding file will be opened.</p> + +<p>Finally, my <code>.bash_profile</code> sets the <code>EDITOR</code> variable to <code>editor</code> (<code>export EDITOR=/usr/bin/editor</code>); this environment variable is what most tools (git included) will use to invoke a text editor.</p> + +<p>On my machine, starting the daemon takes 1.4s. Creating a client windows takes around 0.150s.</p> + +<p>If you want to control the environment in which the daemon process is started, you can launch it explicitly by running <code>emacs --daemon</code>.</p> + +<p>Cool kids use <a href="http://spacemacs.org/">Spacemacs</a> these days, which comes with all sort of convenient settings built in, and I&rsquo;m told that it does daemonization out of the box. I haven&rsquo;t taken the time to give Spacemacs a try yet.</p> + +<p>Finally, sometimes having all editor windows share the same process is not the right thing, because I do stuff which makes Emacs a bit unstable. (It&rsquo;s not very good at asynchronous communication with the rest of the world, so for example accessing a file through SSH from Emacs can hang the process when network goes bad.). I&rsquo;ve been bitten a few times by a crash of all editor windows at the same time, and since then, when I know I&rsquo;m going to do &ldquo;heavy stuff&rdquo;, I launch a separate process for it (just <code>emacs</code> instead of <code>editor</code> or <code>emacsclient</code>).</p> + +<p>P.S.: To precisely measure the startup time, ask Emacs to evaluate a Lisp expression on startup, to kill it immediately.:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span>$ <span class="nb">time</span> emacs --eval <span class="s2">"(save-buffers-kill-terminal)"</span> +$ <span class="nb">time</span> emacsclient -a <span class="s1">&#39;&#39;</span> -c -e <span class="s2">"(save-buffers-kill-terminal)"</span> +</pre></div> +</td></tr></tbody></table> +</div> + + Does anyone still care about printed proceedings? (Grab some at NEU this week!) + + urn:http-prl-ccs-neu-edu:-blog-2016-06-13-does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week + 2016-06-13T10:50:14Z + 2016-06-13T10:50:14Z + + Gabriel Scherer + +<p>Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!</p> +<!-- more--> + +<p>If you are in the area and are interested, feel free to send me an email and come grab them. We have ASPLOS from XIII to XX, PLDI from 2005 to 2015 (but not 2014), and OOPSLA from 2002 to 2015. When I saw the stack, I grabbed the POPL ones from 2002 to 2015, but in fact I have no idea what to do with them and I&rsquo;m a bit skeptical they would be of use to me; if you know you would use them, I would be glad to let you have them.</p> + +<p>If you were to buy those proceedings at conference-subscription rates today, it would cost you a small fortune. Yet nobody seems to want them. An odd disconnect, that I found amusing and maybe worthy of a blog post.</p> + +<p>But don&rsquo;t get me wrong, the future of printed proceedings is not an important question. We should rather be asking ourselves: why are the products of the work of our communities not easily accessible in an Open Access long-term archive? Are you submitting your articles to arXiv, or another archive? Why not?</p> + +<p>Not caring about printed proceedings is perfectly fine; but please care about people outside institutions that want to access your work &mdash; for example, master student myself.</p> + +<hr /> + +<p><em>Update (August 2017):</em> Gabriel returned to France and left the POPL proceedings at his desk. The proceedings are sitting in the hallway at NEU, in case anyone cares.</p> + + ICFP 2016: looking for student volunteers + + urn:http-prl-ccs-neu-edu:-blog-2016-06-07-icfp-2016-looking-for-student-volunteers + 2016-06-07T11:53:47Z + 2016-06-07T11:53:47Z + + Gabriel Scherer + +<p>If you are a student, you should consider <a href="http://goo.gl/forms/fKg3vpjNryBlGWB32">applying</a> to become an ICFP 2016 student volunteer! The deadline for application is July 31st, 2016.</p> +<!-- more--> + +<p><a href="http://conf.researchr.org/attending/icfp-2016/Student+Volunteers">ICFP 2016</a>, the Internal Conference on Functional Programming, is happening in Nara, Japan. If you are a student, you may be interest in being a Student Volunteer: you help run the conference, and in exchange do not pay registration fees &mdash; but you still have to find funding for the travel, hosting, and dinners. Quoting the <a href="http://conf.researchr.org/attending/icfp-2016/Student+Volunteers">Student Volunteer</a> webpage:</p> + +<blockquote> + <p>ICFP is pleased to offer a number of opportunities for student volunteers, who are vital to the efficient operation and continued success of the conference each year. The student volunteer program is a chance for students from around the world to participate in the conferences whilst assisting us in preparing and running the event.</p> + <p>Job assignments for student volunteers include assisting with technical sessions, workshops, tutorials and panels, helping the registration desk, operating the information desk, helping with traffic flow, and general assistance to keep the conferences running smoothly.</p> + <p>The Student Volunteer Program helps more students attend the ICFP conference by covering conference fees (but not travel or lodging expenses) in exchange for a fixed number of work hours (usually from 8 to 12) helping with the conference organization (registration and information desks, assistance during talk sessions, etc.).</p> + <p>The Student Volunteer registration covers:</p> + <ul> + <li>Access to all workshops and the main conference,</li> + <li>Daily lunches and coffee breaks,</li> + <li>Access to social events, including the banquet.</li></ul> + <p>To apply, please fill the <a href="http://goo.gl/forms/fKg3vpjNryBlGWB32">following form</a>.</p> + <p>The application deadline is July 31st, 2016. Applications after this date may be considered pending availability.</p> + <p>You can send questions about student volunteering to <code>icfp-SV at researchr dot org</code>.</p></blockquote> + +<p>I was &ldquo;student volunteer captain&rdquo; at ICFP last year in Vancouver, and I will do it again this year. My entirely personal take on the thing is that being a Student Volunteer is worth it, but that being a Captain is too much work.</p> + +<p>The main downside of being a volunteer is some of the shifts are at the registration desk, and they may imply missing some of the talks &mdash; and also you may have to get up early for your duties. The upsides are many. You get belong to a small group of nice people. You have interactions with many people without much effort; you will enjoy the sparks of gratitude in the eyes of the &ldquo;Where is Workshop Room B2?&rdquo; crowd. You have a small peek into the kind of work involved in running a conference; most people actually working on the organization (we SVs are hobbyists) are pouring surprising amount of work. Also, you learn to fold tee-shirts very well, if you&rsquo;re on &ldquo;bag stuffing&rdquo; duty.</p> + +<p>Being a student volunteer can be combined with other forms of travel support, such as SIGPLAN PAC funding; see the <a href="http://conf.researchr.org/attending/icfp-2016/student-travel-support">travel support page</a> for more details.</p> + +<p>Another thing you should think about is applying to <a href="http://conf.researchr.org/track/icfp-2016/PLMW-ICFP-2016">PLMW</a>, the Programming Languages Mentoring Workshop that happens at ICFP, POPL, and PLDI. PLMW funding covers the whole conference cost (travel, housing, registration, dinners), so if you get PLMW funding you have no financial motivation to be a student volunteer. This year, PLMW focuses on early career graduate students.</p> + + Measuring GC latencies in Haskell, OCaml, Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + 2016-05-24T10:51:34Z + 2016-05-24T10:51:34Z + + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> + + NEPLS on May 31st at UMass, Amherst + + urn:http-prl-ccs-neu-edu:-blog-2016-05-03-nepls-on-may-31st-at-umass-amherst + 2016-05-03T08:21:07Z + 2016-05-03T08:21:07Z + + Gabriel Scherer + +<p>It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!</p> +<!-- more--> + +<blockquote> + <p>The next New England Programming Languages and Systems Symposium will take place on Tuesday, May 31st 2016 at University of Massachusetts, Amherst. Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is Tuesday, May 17th. Send your proposal to talks@nepls.org.</p> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <p> http://www.nepls.org/</p></blockquote> + +<p>I&rsquo;m personally fond of such regional events, which is a great time to learn about the research around us in a less formal and exhausting setting than a 200-attendees conference.</p> + +<p>If you are in the area, please consider applying to talk about your work. If one of your colleague is working on something you find exciting, please invite them to apply!</p> \ No newline at end of file diff --git a/blog/feeds/Author-Gabriel-Scherer.rss.xml b/blog/feeds/Author-Gabriel-Scherer.rss.xml new file mode 100644 index 00000000..ad96cd6e --- /dev/null +++ b/blog/feeds/Author-Gabriel-Scherer.rss.xml @@ -0,0 +1,1211 @@ + + + + PRL Blog: Posts tagged 'Author: Gabriel Scherer' + PRL Blog: Posts tagged 'Author: Gabriel Scherer' + http://prl.ccs.neu.edu/blog/tags/Author-Gabriel-Scherer.html + Sun, 13 Aug 2017 14:29:41 UT + Sun, 13 Aug 2017 14:29:41 UT + 1800 + + Reviews and author responses: we should stop asking for 500-word responses + http://prl.ccs.neu.edu/blog/2017/08/13/reviews-and-author-responses-we-should-stop-asking-for-500-word-responses/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-13-reviews-and-author-responses-we-should-stop-asking-for-500-word-responses + Sun, 13 Aug 2017 14:29:41 UT + Gabriel Scherer + +<p>This year I reviewed many ICFP submissions, and got to be on the receiving end of equally many author responses (also sometimes called, somewhat combatively, rebuttals). I found that there was a large difference between the official written advice on author responses and what I, as a reviewer reading the responses, found effective. In particular, I now believe that limiting yourself to 500 words should strongly be avoided &mdash; we should even stop giving that advice.</p> +<!-- more--> + +<p>This year, I had the honor (and accompanying load of work) of being a Program Committee (PC) member at the ICFP conference. It was my first time being a PC member at a conference, and I found it extremely pleasant and interesting, thanks to the authors who sent us articles to review, my fellow PC members, and the ever-smiling careful balancing work of our PC chair, Mark Jones. It was also a lot of work, starting with 18 reviews to do over a month and a half, an intense PC meeting, and the new &ldquo;second phase&rdquo; process with the opportunity for authors and reviewers to exchange feedback on changes requested by the program committee.</p> + +<p>There is little guidance on how to write author responses, although this <a href="http://www.pl-enthusiast.net/2014/09/17/advice-writing-author-response/">blog post</a> by Michael Hicks on pl-enthusiast is quite good. One thing that is obvious as a reviewer and is only slightly brushed in this post, however, is that author responses should <em>not</em> aim to fit a 500 words limit, and in fact I believe that it is a bad idea to do so.</p> + +<p>As for most conference, the ICFP review system recommends (in writing) to authors to keep their response to 500 words (some systems also highlight words after those in red to make the point clear). Don&rsquo;t do this! The least convincing responses I have seen are those that followed this recommendation.</p> + +<p>(I have also seen at least 18*2 other reviewers read the same responses I read, most of them well over 500 words, and <em>none</em> of them made any comment on the length of the author responses.)</p> + +<p>We have a frustrating situation where the explicit rule is different from the thing people do in practice. This is bad for newcomers that do not know the norms and cannot tell if ignoring the rule may hurt them. This is the point of this blog post:</p> + +<ul> + <li> + <p>If you are an author, please know that disrespecting the 500-words limit is the <em>right thing</em> to do. You should also know, of course, that people have limited time, so keep the main body of your response reasonably long. (I spent a day and a half on your paper already, I am willing to spend 10 additional minutes reading the response.)</p></li> + <li> + <p>If you are a program committee chair or a Magical HotCRP Wizard, please remove this silly recommendation to keep responses to 500 words. (See an alternative proposal at the end of this post.)</p></li></ul> + +<h2 id="my-personal-response-format">My personal response format</h2> + +<p>My author responses start with general comments that elaborate on the main point I want to tell all reviewers. Then, a second part contains per-reviewer comments (one section per reviewer); it is clearly marked as skippable. Here is the skeleton of the last response I wrote:</p> + +<blockquote> + <p>We thank the reviewers for their work on our article and their detailed feedback. We start with a general discussion that responds to the salient points raised by reviewers. In a second part, we provide detailed responses to the questions/remarks of each reviewer.</p> + <h3 id="general-discussion">General discussion</h3> + <p>[..]</p> + <h3 id="specific-questionscomments-review-a">Specific questions/comments: review #A</h3> + <p>[..]</p> + <h3 id="specific-questionscomments-review-b">Specific questions/comments: review #B</h3> + <p>[..]</p> + <h3 id="specific-questionscomments-review-c">Specific questions/comments: review #C</h3> + <p>[&hellip;]</p></blockquote> + +<p>For this particular response, the &ldquo;General discussion&rdquo; section used 1296 words according to <code>wc -w</code> (M-x shell-command-on-region). In the following sections, I quote the reviews to answer specific points, email-style (following the Markdown syntax that HotCRP renders properly).</p> + +<h2 id="suggested-wording-for-pc-chairs">Suggested wording for PC chairs</h2> + +<p>If you are a PC chair, you should remove the suggestion of respecting a 500 words limit for your conference. Here would be a suggested alternative wording:</p> + +<blockquote> + <p>Please remember, in writing your author response, that reviewers may stop reading the response at any point. We suggest having a reasonably-sized main section where you make your most important high-level comments, and clearly marked sections where you answer individual reviewer&rsquo;s questions. It is not useful nor productive to answer every point of each review, you should focus on the comments that you believe the reviewers are most interested in.</p></blockquote> + + Syntactic parametricity strikes again + http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-syntactic-parametricity-strikes-again + Mon, 05 Jun 2017 14:27:44 UT + Gabriel Scherer, Li-Yao Xia + +<p>In this blog post, reporting on a collaboration with <a href="https://poisson.chat/">Li-Yao Xia</a>, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely &ldquo;syntactic&rdquo; way, by looking at the structure of canonical derivations. More precisely, I prove that \( +\newcommand{\List}[1]{\mathsf{List}~#1} +\newcommand{\Fin}[1]{\mathsf{Fin}~#1} +\newcommand{\Nat}[1]{\mathbb{N}} +\newcommand{\rule}[2]{\frac{\displaystyle \array{#1}}{\displaystyle #2}} +\newcommand{\judge}[2]{{#1} \vdash {#2}} +\newcommand{\emptyrule}[1]{\begin{array}{c}\\[-1em] #1 \end{array}} + ∀α. \List α → \List \alpha +\) is isomorphic to \( + Π(n:\Nat{}). \List{(\Fin{n})} +\) where \(\Fin{n}\) is the type of integers smaller than \(n\), corresponding to the set \(\{0, 1, \dots, n-1\}\).</p> +<!-- more--> + +<p>Context: Last week I had the pleasure of visiting UPenn, where I had many interesting discussions with various people. It was also an occasion to temporarily resume a discussion/collaboration I have with Li-Yao Xia, who is currently an intern there, and Jean-Philippe Bernardy, about testing polymorphic programs and its relation to canonical representations for System F.</p> + +<p>During one of our blackboard discussion, Li-Yao and I did a manual proof of a cool result: we proved a parametricity theorem for \(∀α. \List α → \List α\) using syntactic methods, namely proof search among canonical proofs. (This is an idea that I have been playing with since the last year of my <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/">PhD thesis</a>, where I unsuccessfully tried to extend my work on canonical forms for the simply-typed lambda-calculus to polymorphism. It is here worked out on an specific example, but my end goal is to turn the manual reasoning into an algorithm.)</p> + +<p>You may wonder, first, why the isomorphism holds. The idea is that a polymorphic function of type \(\List α → \List α\) cannot inspect the elements of the input list; it can only use them in the resulting list, possibly duplicating, reordering or dropping some elements. On any input list of size \(n\), the behavior of the function can be described by a list of indices in \([0; n-1]\). For example, if the input \([x, y, z]\) (for some values of \(x, y, z\)) gives the output \([y, y, x]\), then this relation will hold on <em>any</em> value of \(x, y, z\), as the function cannot inspect their value or even test them for equality. The behavior of this function on lists of size 3 can be fully described by the list of indices \([1, 1, 0]\). Its whole behavior is then uniquely determined by one such list for each possible size:</p> + +<p>\[ + ∀α. \List α → \List α \quad≃\quad Π(n:\Nat{}). \List{(\Fin n)} +\]</p> + +<p>The idea behind the &ldquo;syntactic&rdquo; (proof-theoretic?) proof method is the following: the set of closed values at a type \(A\) is isomorphic to the <em>search space</em> for canonical/normal derivations of \(\judge{}{A}\). We have tools (in particular the notion of <em>invertible</em> inference rules) to reason on those – in this post I will only present the reasoning informally, but it can easily be made formally precise.</p> + +<p>We start by looking at the shape of the search space for</p> + +<p>\[ + \judge{}{∀α. \List α → \List α} +\] or, said otherwise, of the judgment \[ + \judge{}{\List α → \List α} +\]</p> + +<p>with a fresh/abstract type variable \(α\). (I will not be keeping opened type variables in context to avoid confusing them with hypotheses.)</p> + +<p>Any derivation of a function type, without loss of generality (w.l.o.g), is equivalent to a derivation starting with a function introduction. This is the η-expansion rule for functions: any proof term \(e\) is equivalent to \(λx.~(e~x)\), a proof that starts with a \(λ\). So any proof can be taken to start as follows: \[ +\rule{ +\judge{\List \alpha}{\List \alpha} +}{ +\judge{}{\List \alpha \to \List \alpha} +} +\] we can, w.l.o.g, unfold the recursive type in the context (\(\List α = 1 + (α × \List α)\)): \[ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}} +\]</p> + +<p>A derivation with a sum type as hypothesis can, w.l.o.g, be assumed to start by splitting on this pair (this is the η-expansion rule for sums): \[ +\rule{ +\judge{1}{\List α} +\quad +\judge{α × \List α}{\List α} +}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]</p> + +<p>In the right subgoal, we can always, w.l.o.g, split a hypothesis of product type: \[ +\rule{ +\emptyrule{\judge{1}{\List α}} +\quad +\rule{ +\judge{α, \List α}{\List α} +}{ +\judge{α × \List α}{\List α} +}}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]</p> + +<p>Now, an interesting pattern emerges. In the process of trying to prove \(\judge{\List α}{\List α}\), we have to prove the (right) subgoal \(\judge{α,\List α}{α}\). We can generalize this derivation by assuming that we start with some number \(n\) of variables of type \(α\) in the context (we write \(α^n\) for this): \[ +\rule{ +\rule{ +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n, 1}{\List \alpha} +} +\quad +\rule{ +\judge{\alpha^{n+1}, \List \alpha}{\List \alpha} +}{ +\judge{\alpha^n, \alpha \times \List \alpha}{\List \alpha} +}}{ +\rule{ +\judge{\alpha^n, 1 + (\alpha \times \List \alpha)}{\List \alpha} +}{ +\judge{\alpha^n, \List \alpha}{\List \alpha} +}} +\]</p> + +<p>\[ +\newcommand{\llbracket}{〚} +\newcommand{\rrbracket}{〛} +\newcommand{\sem}[1]{\llbracket{} #1 \rrbracket{}} +\]</p> + +<p>Let us write \(\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}}\) for the search space corresponding to all possible derivations of the judgment \(\judge{\alpha^n, \List \alpha}{\List \alpha}\). All the proof steps above have been done &ldquo;without loss of generality&rdquo; (in terms of focusing, we only used invertible rules), so they appear in any such derivation. Similarly, let us write \(\sem{\judge{\alpha^n}{\List \alpha}}\) for the space of all possible derivations of \(\judge{\alpha^n}{\List \alpha}\), then above we have proven that \[ +\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} +\quad=\quad +\sem{\judge{\alpha^n}{\List \alpha}} +\times +\sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} +\]</p> + +<p>This equality can be unfolded at will \[ +\begin{align} +&amp; \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+3}, \List \alpha}{\List \alpha}} \\ += &amp; \dots \\ +\end{align} +\]</p> + +<p>or written as an infinite product \[ + \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} + \quad=\quad + \prod_{k \in \Nat{}}{\sem{\judge{\alpha^{n+k}}{\List \alpha}}} +\] and, in particular, \[ +\begin{align} +&amp; \sem{\judge{}{\List \alpha \to \List \alpha}} \\ += &amp; \sem{\judge{\alpha^0, \List \alpha}{\List \alpha}} \\ += &amp; \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ +\end{align} +\]</p> + +<p>Now let&rsquo;s look at the structure of the derivations of \(\judge{\alpha^n}{\List \alpha}\). A proof of this judgment cannot start with a &ldquo;left rule&rdquo;, inspecting the value of one of the \(n\) variables of type \(α\), given that the structure of \(α\) is unknown/abstract. It must start by choosing to either build the empty list or a cons cell. We write this as follows (after unfolding the type):</p> + +<p>\[ +\rule{ +\rule{ +\judge{\alpha^n}{1} +\quad\oplus\quad +\judge{\alpha^n}{\alpha \times \List \alpha} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]</p> + +<p>The \(\oplus\) notation between two judgments is non-standard; it means that they are not two requirements of the same proof, but two alternatives for possible proofs. All valid proofs fit that structure, and they either have a \(\judge{\alpha^n}{1}\) premise or a \(\judge{\alpha^n}{\alpha \times \List \alpha}\) premise. With this syntax, we are describing a set of possible derivations, rather than a single (partial) derivation.</p> + +<p>Proofs of \(\judge{\Gamma}{1}\) are trivial, and a proof of a product is always, w.l.o.g, a product of proofs (in intuitionistic logic / the λ-calculus they reuse the same context), so we can decompose further: \[ +\rule{ +\rule{ +\rule{ +}{ +\judge{\alpha^n}{1} +} +\quad\oplus\quad +\rule +{ +\judge{\alpha^n}{\alpha} +\quad +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n}{\alpha \times \List \alpha} +} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]</p> + +<p>There is exactly one possible proof of \(\judge{\alpha^n}{1}\), so its search space is \(1\), the unit set (with a single element). There are exactly \(n\) possible proofs of \(\judge{\alpha^n}{\alpha}\), so the search space is just \(n\), seen as a set, or, in type-theoretic notation, \(\Fin{n}\). We thus have the recursive equation: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +1 + (\Fin n \times \sem{\judge{\alpha^n}{\List \alpha}}) +\]</p> + +<p>This type is either \(1\), or a \(\Fin{n}\) and itself, recursively. This is exactly a list: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +\List{(\Fin{n})} +\]</p> + +<p>so, plugging everything together: \[ +\begin{align} +&amp; \sem{\forall \alpha. \List \alpha \to \List \alpha} \\ += &amp; \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ += &amp; \prod_{n \in \Nat{}}{\List{(\Fin{n})}} \\ +\end{align} +\]</p> + +<h3 id="post-scriptum">Post Scriptum</h3> + +<p>Some of reasoning steps above can be formulated in a way that is less clear but more familiar, as a sequence of type isomorphisms. For example, the first part on \(\sem{\judge{\alpha^n, \List +\alpha}{\List \alpha}}\) can written as:</p> + +<p>\[ +\begin{align} +&amp; +∀α. αⁿ × \List α → \List α +\\ &amp; += \text{(unfold List)} +\\ &amp; + ∀α. αⁿ × (1 + α × \List α) → \List α +\\ &amp; + = \text{(distribute × over +)} +\\ &amp; + ∀α. ((αⁿ × 1) + (αⁿ⁺¹ × \List α)) → \List α +\\ &amp; + = \text{(A × 1 ≃ A)} +\\ &amp; + ∀α. (αⁿ + (αⁿ⁺¹ × \List α)) → \List α +\\ &amp; + = \text{(A+B) → C ≃ (A→C)×(B→C)} +\\ &amp; + ∀α. (αⁿ → \List α) × (αⁿ⁺¹ × \List α → \List α) +\\ &amp; + = \text{(distribute ∀α below product)} +\\ &amp; + (∀α. αⁿ → \List α) × (∀α. αⁿ⁺¹ × \List α → \List α) +\\ +\end{align} +\]</p> + +<p>Reading this equational sequence, it may look like we had to make the right choice at each step; but the proof-search perspective reveals that there were in fact no choices, as each time we apply invertible rules (&ldquo;w.l.o.g. rules&rdquo;).</p> + +<p>Furthermore, some parts cannot be derived in this style; in the latter part of the proof, the isomorphism between \(∀\alpha. \alpha^n → \alpha\) and \(\Fin{n}\), which is immediate from a proof search perspective, cannot be justified in this way. (In particular, \(A×B → C\) is <em>not</em> isomorphic to \((A→C)+(B→C)\).)</p> + +<h3 id="going-further">Going further</h3> + +<ul> + <li> + <p>It is an unfortunately-little-known obvious fact that many things we associate to &ldquo;free theorems&rdquo; can be recovered by proof search. For example, it is much simpler to prove that the only inhabitant of \(\forall \alpha. \alpha \to \alpha\) is the identity using proof search than parametricity. I briefly discussed the idea in the section 1.3 of my 2015 article, <a href="http://gallium.inria.fr/~scherer/research/unique_inhabitants/unique_stlc_sums-long.pdf">Which simple types have a unique inhabitant?</a>.</p></li> + <li> + <p>If you are unfamiliar with proof search (or the LF community) and curious about what I mean by &ldquo;canonical forms&rdquo; and why I think this is an important idea, see my non-technical 2017 article <a href="http://www.ccs.neu.edu/home/gasche/research/canonical-forms/snapl.pdf">Search for Program Structure</a>. The problem of extending the notion of canonical forms to arbitrary polymorphic types is briefly discussed in the section 14.5 of my 2016 <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">phd manuscript</a>.</p></li> + <li> + <p>If you haven&rsquo;t heard of it yet, you would probably be interested in the 2010 article <a href="http://publications.lib.chalmers.se/records/fulltext/local_99387.pdf">Testing Polymorphic Properties</a> by Jean-Philippe Bernardy, Patrik Jansson and Koen Claessen. Li-Yao has a 2016 implementation called <a href="https://github.com/Lysxia/metamorph">Metamorph</a> that got us talking together. The problems of understanding canonical forms and testing are quite related, but yet not exactly the same&hellip;</p></li></ul> + +<h3 id="you-might-also-like">You might also like</h3> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="https://williamjbowman.com/blog/2017/01/03/toward-type-preserving-compilation-of-coq-at-popl17-src/">Toward Type-Preserving Compilation of Coq, at POPL17 SRC</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> + + PRL at SNAPL'17 + http://prl.ccs.neu.edu/blog/2017/04/25/prl-at-snapl-17/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-04-25-prl-at-snapl-17 + Tue, 25 Apr 2017 16:46:54 UT + Gabriel Scherer + +<p>PRL recently produced three papers for the <a href="http://snapl.org/2017/index.html">SNAPL</a> conference.</p> + +<ul> + <li><a href="https://dbp.io/pubs/2017/linking-types-snapl.pdf">Linking Types for Multi-Language Software: Have Your Cake and Eat It Too</a>, by Daniel Patterson and Amal Ahmed.</li> + <li><a href="http://www.ccs.neu.edu/home/gasche/research/canonical-forms/snapl.pdf">Search for Program Structure</a>, by Gabriel Scherer</li> + <li><a href="http://www.ccs.neu.edu/racket/pubs/typed-racket.pdf">Migratory Typing: Ten Years Later</a>, by Sam Tobin-Hochstadt, Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Ben Greenman, Andrew M. Kent, Vincent St-Amour, T. Stephen Strickland and Asumu Takikawa</li></ul> +<!-- more--> + +<h3 id="httpsdbpiopubs2017linking-types-snaplpdflinking-types-for-multi-language-software-have-your-cake-and-eat--it-too"><a href="https://dbp.io/pubs/2017/linking-types-snapl.pdf">Linking Types for Multi-Language Software: Have Your Cake and Eat It Too</a></h3> + +<p>Daniel Patterson and Amal Ahmed, 2017</p> + +<blockquote> + <p>Software developers compose systems from components written in many different languages. A business-logic component may be written in Java or OCaml, a resource-intensive component in C or Rust, and a high-assurance component in Coq. In this multi-language world, program execution sends values from one linguistic context to another. This boundary-crossing exposes values to contexts with unforeseen behavior—that is, behavior that could not arise in the source language of the value. For example, a Rust function may end up being applied in an ML context that violates the memory usage policy enforced by Rust’s type system. This leads to the question of how developers ought to reason about code in such a multi-language world where behavior inexpressible in one language is easily realized in another.</p> + <p>This paper proposes the novel idea of linking types to address the problem of reasoning about single-language components in a multi-lingual setting. Specifically, linking types allow programmers to annotate where in a program they can link with components inexpressible in their unadulterated language. This enables developers to reason about (behavioral) equality using only their own language and the annotations, even though their code may be linked with code written in a language with more expressive power.</p></blockquote> + +<h3 id="httpwwwccsneueduhomegascheresearchcanonical-formssnaplpdfsearch-for-program-structure"><a href="http://www.ccs.neu.edu/home/gasche/research/canonical-forms/snapl.pdf">Search for Program Structure</a></h3> + +<p>Gabriel Scherer, 2017.</p> + +<blockquote> + <p>The community of programming language research loves the Curry-Howard correspondence between proofs and programs. Cut-elimination as computation, theorems for free, &lsquo;call/cc&rsquo; as excluded middle, dependently typed languages as proof assistants, etc.</p> + <p>Yet we have, for all these years, missed an obvious observation: &ldquo;the structure of <em>programs</em> corresponds to the structure of proof <em>search</em>&rdquo;. For pure programs and intuitionistic logic, more is known about the latter than the former. We think we know what programs are, but logicians know better!</p> + <p>To motivate the study of proof search for program structure, we retrace recent research on applying the logical technique of focusing to study the canonical structure of simply-typed λ-terms. We then motivate the open problem of extending canonical forms to support richer type systems, such as polymorphism, by discussing a few enticing applications of more canonical program representations.</p></blockquote> + +<h3 id="httpwwwccsneueduracketpubstyped-racketpdfmigratory-typing-ten-years-later"><a href="http://www.ccs.neu.edu/racket/pubs/typed-racket.pdf">Migratory Typing: Ten Years Later</a></h3> + +<p>Sam Tobin-Hochstadt, Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Ben Greenman, Andrew M. Kent, Vincent St-Amour, T. Stephen Strickland and Asumu Takikawa, 2017.</p> + +<blockquote> + <p>In this day and age, many developers work on large, untyped code repositories. Even if they are the creators of the code, they notice that they have to figure out the equivalent of method signatures every time they work on old code. This step is time consuming and error prone.</p> + <p>Ten years ago, the two lead authors outlined a linguistic solution to this problem. Specifically they proposed the creation of typed twins for untyped programming languages so that developers could migrate scripts from the untyped world to a typed one in an incremental manner. Their programmatic paper also spelled out three guiding design principles concerning the acceptance of grown idioms, the soundness of mixed-typed programs, and the units of migration.</p> + <p>This paper revisits this idea of a migratory type system as implemented for Racket. It explains how the design principles have been used to produce the Typed Racket twin and presents an assessment of the project’s status, highlighting successes and failures.</p></blockquote> + +<p>.</p> + +<p>SNAPL is not dissimilar to the (french-speaking) <a href="http://jfla.inria.fr/">JFLA</a> that I am more familiar with &mdash; with an added irritating call for paper and unreasonable registration price. It has an interesting diversity of topics of presentation: see also the complete <a href="http://snapl.org/2017/papers.html">list of accepted papers</a> this year, and the <a href="http://snapl.org/2015/papers.html">list of the previous edition</a>.</p> + + Bullets are good for your Coq proofs + http://prl.ccs.neu.edu/blog/2017/02/21/bullets-are-good-for-your-coq-proofs/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-21-bullets-are-good-for-your-coq-proofs + Tue, 21 Feb 2017 19:04:28 UT + Gabriel Scherer + +<p>I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.</p> + +<p>Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.</p> +<!-- more--> + +<h2 id="why-bullets">Why bullets</h2> + +<p>While you are doing a proof, Coq shows a list of subgoals that have to be proved before the whole proof is complete. Most proof steps will operate on the current active subgoal, changing the hypotheses or the goal to prove, but some proof steps will split it into several subgoals (growing the total list of goals), or may terminate the proof of the current subgoal and show you the next active subgoal.</p> + +<p>Before bullets, a typical proof script would contain the proofs of each subgoal, one after another.</p> + +<pre><code>induction foo. (* this creates many subgoal *) + +proof of first subgoal. + +proof of second subgoal.</code></pre> + +<p>There are many ways to structure this to make the structure more apparent: people would typically have a comment on each subgoal, or make disciplined use of indentation and blank lines. But, in my experience, a major problem with this style was maintainability in the face of changes to the definitions or parts of automation. It could be very hard of what was happening when a proof suddenly broke after a change before in the file:</p> + +<ul> + <li> + <p>If a proof step now proves <em>less</em> things, then what used to be the end of a subgoal may not be anymore. Coq would then start reading the proof of the next subgoal and try to apply it to the unfinished previous goals, generating very confusing errors (you believe you are in the second subgoal, but the context talks about a leaf case of the first goal).</p></li> + <li> + <p>If a proof step now proves <em>more</em> things, it is also very bad! The next proof steps, meant for the first subgoal (for example), would then apply to the beginning of the second subgoal, and you get very confusing errors again.</p></li></ul> + +<p>What we need for robustness is a way to indicate our <em>intent</em> to Coq, when we think that a subgoal is finished and that a new subgoal starts, so that Coq can fail loudly at the moment where it notices that this intent does not match reality, instead of at an arbitrary later time.</p> + +<p>(The <code>S*Case</code> tactics used in (older versions of) Software Foundations can solve this problem if used in a carefully, systematic way, and additionally provides naming. Alexandre Pilkiewicz implemented an even more powerful <a href="https://github.com/pilki/cases">cases</a> plugin. Bullets are available in standard Coq since 8.4 (released in 2012), and can be used with no effort.)</p> + +<p>There is not much discussion of bullets around; see the <a href="https://coq.inria.fr/distrib/8.6/refman/Reference-Manual009.html#sec326">documentation</a> in the Coq manual. I learned a lot from Arthur Azevedo de Amorim&rsquo;s <a href="https://github.com/arthuraa/poleiro/blob/master/theories/Bullets.v">Bullets.v</a> file.</p> + +<p>Finally, some people don&rsquo;t use bullets, because they systematically use so much automation that they never see subgoals &mdash; each lemma has a one-line proof. This is also a valid style. (I have been going to Adam Chlipala&rsquo;s <a href="https://frap.csail.mit.edu/main">Formal Reasoning about Programs</a> 2017 class, where Adam ignores bullets because that is his usual style.) Because I am not crushy enough to do this from the start, my proofs tend to start with cases and subgoals, and then I refine them to add more automation for robustness. I found bullets very useful for the first step, and during the refinement process.</p> + +<h2 id="bullets">Bullets</h2> + +<p>Bullets are actually a combination of two features, braces <code>{ ... }</code> and actual list bullets &mdash; <code>-</code>, <code>+</code>, <code>*</code>, or homogeneous repetitions of those, for example <code>--</code> or <code>***</code>.</p> + +<h3 id="braces">Braces</h3> + +<p>The opening brace <code>{</code> focuses the proof on the current subgoal. If you finish the proof of the subgoal, the following subgoal will not become accessible automatically; you have to use the closing brace <code>}</code> first. (If you finish the goal earlier than you think, you get an error.) Conversely, <code>}</code> fails if the subgoal is not complete. (If you fail to finish, you get an error.)</p> + +<p>The previous example can thus be written as follows, and will be more robust:</p> + +<pre><code>induction foo. (* this creates many subgoal *) +{ + proof of first subgoal. +} +{ + proof of second subgoal. +}</code></pre> + +<p>If you also want to make sure that an error occurs if the number of subgoals changes (for example if new constructors are added to the inductive type of <code>foo</code>), you can use an outer layer of braces:</p> + +<pre><code>{ induction foo. (* this creates many subgoal *) + { + proof of first subgoal. + } + { + proof of second subgoal. + } +} (* would fail if a new subgoal appeared *)</code></pre> + +<h3 id="list-bullets">List bullets</h3> + +<p>A bullet, for example <code>--</code>, also focuses on the next subgoal. The difference is that when the subgoal is finished, you do not have a closing construction, you must use the same bullet to move to the next subgoal. (Again, this fails if the first proof step changes to prove too much or too little.) With bullets you would write</p> + +<pre><code>induction foo. (* this creates many subgoal *) ++ proof of first subgoal. ++ proof of second subgoal.</code></pre> + +<p>Bullets can be nested, but you must use different bullets for the different nesting levels. For example, if this proof is only one subgoal of a larger proof, you can use:</p> + +<pre><code>- induction foo. (* this creates many subgoal *) + + proof of first subgoal. + + proof of second subgoal. +- (* would fail if a new subgoal appeared *) + rest of the proof</code></pre> + +<p>The natural ordering of bullets, I think, is by increasing number of lines: <code>-</code>, <code>+</code> then <code>*</code> (and then multi-character bullets, I guess). You can also mix bullets with braces: the opening brace resets the bullet scope, any bullet can be used again with the subgoal.</p> + +<p>This gives a large space of freedom in how you want to use these features. You can use only braces, only bullets, braces and only one level of bullets, etc. My own style evolved with experience using the feature, and I will present the current status below.</p> + +<h2 id="my-current-bullet-style">My current bullet style</h2> + +<p>When deciding how to use bullets, one distinguishes the commands that preserve the number of subgoals and those that may create new subgoals. I use some additional distinctions.</p> + +<p>Some tactics, for example <code>assert</code>, create a number of subgoals that is <em>statically</em> known, always the same for the tactic. I then use braces around each sub-proof, except the last one, which I think of as the &ldquo;rest&rdquo; of the current proof.</p> + +<pre><code>assert foo as H. +{ proof of foo. } +rest of the proof using H:foo.</code></pre> + +<p>(If the proof of <code>foo</code> takes several lines, I two-indent them, with the braces alone on their lines.)</p> + +<p>Most tactics create a <em>dynamic</em> number of subgoals, that depends on the specifics of the objects being operated on; this is the case of <code>case</code>, <code>destruct</code>, <code>induction</code> for example. In this case, I open a brace before the tactic, and use a bullet for each subgoal.</p> + +<pre><code>{ induction foo; simpl; auto. +- proof of first remaining subgoal. +- proof of second remaining subgoal. + rest of the proof of the second subgoal. +}</code></pre> + +<p>(Notice that the subgoal-creating step is vertically aligned with the proof steps: I use both braces and bullets, but take only one indentation level each time.)</p> + +<p>As an exception, I may omit the braces if we are at the toplevel of the proof (<code>Proof .. Qed</code> serve as braces).</p> + +<p>Note that omitting the braces here and using different bullets when you nest is also just fine. In my experience it gives proofs that are a bit more pleasant to read but also a bit more cumbersome to edit and move around.</p> + +<p>Finally, a not-uncommon mode of use of &ldquo;dynamic&rdquo; tactics in the sense above is to expect all the cases, except one, to be discharged by direct automation (for example they are all absurd except one). When it is my intent that all cases but one be discharged (and not a coincidence), I express it by not using braces (this command preserves the number of subgoals), but marking the remaining subgoal with a new bullet <em>without</em> increasing the indentation level.</p> + +<pre><code>{ induction foo. +- first subgoal. +- second subgoal. + case blah; discharge all sub-subgoals but one. ++ remaining sub-subgoal of the second subgoal. + finish the sub-subgoal. +- third subgoal. +}</code></pre> + +<p>(This is the only time where I use several bullet levels.)</p> + +<p>If you are the kind of programmer that is passionate about indentation style, I should now have tricked you to use bullets to propose a different variant. Otherwise, please consider using bullets anyway, for example by following the style above, it will make your life easier in the face of changing developments.</p> + + Measuring the submission/review balance + http://prl.ccs.neu.edu/blog/2016/12/17/measuring-the-submission-review-balance/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-12-17-measuring-the-submission-review-balance + Sat, 17 Dec 2016 16:33:10 UT + Gabriel Scherer + +<p>How do researchers know whether they are doing &ldquo;enough&rdquo; or &ldquo;too many&rdquo; reviews? A measurable goal is to be review-neutral: to have demanded, through our submissions, as many reviews as we have produced as reviewers.</p> +<!-- more--> + +<h3 id="reviewing-is-good">Reviewing is good</h3> + +<p>I like to review academic papers. It is a very rewarding activity in many different ways. One gets to serve the academic community, helping it function smoothly. One gets a chance at acquiring a much better understanding of someone else&rsquo;s work than idle paper-skimming allows. One gets to send feedback to our colleagues and help them improve their work and its presentation &mdash; it is also an essential way in which we can participate to the formation of student researchers all over the world. Finally, doing reviews helped me develop the skill the judge someone else&rsquo;s work and of forcing oneself to come up with a decisive opinion &mdash; it is surprisingly difficult and only comes with training.</p> + +<p>Doing reviews is also fairly time-consuming. I noticed that the time I spend on each review is generally stable (excursions into previous or related work excluded): around one day and a half for conference reviews, and at least twice more for journal reviews &mdash; I&rsquo;m sure other people have wildly different figures, but I would expect it to be a noticeable time commitment in any case. (Workshop reviews are much easier, at least for the formats I have seen of 2-page extended abstracts, I&rsquo;d say one hour per review.)</p> + +<h3 id="how-many-reviews">How many reviews?</h3> + +<p>Because it is so time-consuming, deciding whether to say &ldquo;yes&rdquo; or &ldquo;no&rdquo; to invitations to review a new paper is not easy: in general I want to say &ldquo;yes&rdquo; (unless I can tell that I will not enjoy reading the paper at all), but it is not reasonable to say &ldquo;yes&rdquo; all the time, because I also need to spend time on other things. When should I say &ldquo;no&rdquo; because I have done &ldquo;too many&rdquo; reviews already?</p> + +<p>We can count the number of reviews that we have done, and we can also estimate the number of reviews that we have demanded of others through our submissions. A natural goal for researchers is to produce at least as many reviews as they demand; if everyone reached this goal, the peer-review system would be at equilibrium without imposing too much of a workload on anyone.</p> + +<p>To estimate the number of reviews a researcher demanded from their peers, you can sum, for each of their submissions to a peer-reviewed venue, the number of reviews that they received, divided by the total number of authors of the submissions.</p> + +<p>\[ \sum_{p \in \mathtt{Submissions}} \frac{\mathtt{reviews}(p)}{\mathtt{authors}(p)} \]</p> + +<p>Out of curiosity, I just measured this balance for myself: over my years doing research I have &ldquo;demanded&rdquo; 10 workshop reviews and 28.5 conference reviews, and &ldquo;produced&rdquo; 6 workshop reviews and 17 conference reviews. If you think that an article would interest me, you shouldn&rsquo;t feel bad about asking me to review it, for now. (On the other hand, my balance <em>this year</em> is positive, so I wouldn&rsquo;t feel to bad about refusing if I had to.)</p> + +<p>Of course, a researcher&rsquo;s balance is highly dependent on where they are in their academic career &mdash; maybe more so that on their personal choices. Students are supposed to submit articles, but are offered few opportunities for doing reviews. When they are invited to do reviews, it is often as sub-reviewer, one review at a time. More established researchers participate in program committees, where they have to do a fair amount of reviews at once &mdash; ten to twenty can be typical in Programming Languages conferences. This means that one naturally starts with a deficit of reviews, and that the opportunity to become balanced or positive only comes over the years.</p> + +<p>(There is much more that could be said about the dynamics of the submission/review balance. I think the idea that a single person should be neutral should not be taken too seriously, because the dynamics are so complex. For example, some people stop doing reviews with a negative balance (students going to the industry for example), so long-time researchers necessarily have a <em>very positive</em> balance that may make short-time researchers balance considerations mostly irrelevant. Another thing is that there is no point doing more reviews than required by the submission flow, and that doing more reviews would build up more reviewing debt under this neutrality criterion &mdash; you can never have everyone positive.)</p> + +<h3 id="quality">Quality</h3> + +<p>This is only a comment on the quantitative aspects of reviewing. Much more important is the qualitative part: are the reviews you receive and produce good reviews? (There is no objective definition of what a good review is; I like reviews that are constructive, help improve the work and its presentation, and catch mistakes.) For a given paper, one or a few very good reviews is more helpful than many bad reviews, so one should not compromise on the quality of one&rsquo;s reviews in order to reach a quantitative goal.</p> + +<h3 id="advice-for-students">Advice for students?</h3> + +<p>While proof-reading this post (thanks!), Ben asked some questions that may be of interest to others &mdash; mostly students, I suppose.</p> + +<blockquote> + <p>If I want to be review-neutral, but I have to accumulate a &ldquo;review debt&rdquo; before I can start reviewing, does this mean I should accept my first opportunity to review and every one that follows (until I&rsquo;m neutral)?</p></blockquote> + +<p>The answer is of course &ldquo;no&rdquo;: one should never feel forced to accept reviews. On the other hand, I do think that it is worthwhile for PhD students to take advantage of the reviews they are offered, so &ldquo;saying yes most of the time&rdquo; sounds like a reasonable strategy to me &mdash; this is just a personal opinion. Some reasons:</p> + +<ul> + <li> + <p>Reviewing is hard and takes training, I think it is good to start practicing early. Students are in a good situation to exercise their reviewing skills at a fairly calm peace (you won&rsquo;t get many reviews anyway), and with more time than more senior people.</p></li> + <li> + <p>Student reviews are often done as sub-reviewer: someone does a review, but also asks for your opinion and includes your sub-review in their review. It is a low-pressure way to do your first reviews, and the ability to exchange opinions with the other reviewer and discuss both reviews is really helpful. Students can also ask for feedback on their reviews to their advisor, which is also very helpful.</p></li> + <li> + <p>Reviewing teaches a few useful things about writing papers as well &mdash; it&rsquo;s always easier to recognize the flaws in others&rsquo; work.</p></li></ul> + +<p>On the other hand, I think you should not accept reviews at times when you cannot invest enough work in the review, or when doing so would be detrimental to you &mdash; whether you are on a deadline, or under too much pressure, or have conflicting commitments, etc. This is more important than anything about a submission/review balance.</p> + +<blockquote> + <p>Do you have any ideas for how young researchers / new researchers can reduce their &ldquo;review footprint&rdquo;? For example, is it possible to volunteer for reviews?</p></blockquote> + +<p>Yes, you can volunteer for reviews by telling the colleagues in your lab that you would be interested in doing reviews and that they should consider giving you some.</p> + +<p>(With the increased use of double-blind submission processes, it is becoming more difficult to pass conference reviews to external researchers. This means that students are relatively unlikely to receive review offers from outside their close colleagues.)</p> + +<p>Besides doing more reviews, the two other factors one could in theory play with are: submitting less papers, and having more co-authors. I think there is something to be said for the first one: one reason to not submit unfinished, buggy or topically-inappropriate articles is that it has a review cost. The second factor should not be considered, I think: &ldquo;did this person contribute to the work?&rdquo; should weight infinitely more for co-authorship decisions.</p> + +<p>Note: Another thing you can ask for is <em>reading reviews other people received</em>. I think that reading reviews is also very helpful for research beginners &mdash; whether reviews of one&rsquo;s own work or someone else&rsquo;s. In particular, I wouldn&rsquo;t know how to write reviews if I hadn&rsquo;t had the opportunity to read reviews before that. If someone you are close to receives reviews, you should consider asking them whether you could have a look.</p> + +<blockquote> + <p>Is being a student volunteer at a conference equal to &ldquo;one review&rdquo;?</p></blockquote> + +<p>I think it is a distinct form of academic service. I don&rsquo;t know how to measure the &ldquo;conference organization cost&rdquo; we impose to our academic colleagues. (If there are around 500 attendants to a typical Programming Languages conference, it means that for every 500 conferences you attend you should organize one all by yourself.)</p> + + SRC-submissions + http://prl.ccs.neu.edu/blog/2016/11/17/src-submissions/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-17-src-submissions + Thu, 17 Nov 2016 13:52:52 UT + Gabriel Scherer + +<p>Max New, Daniel Patterson and Ben Greenman recently wrote three two-page abstracts on what they are working on right now. Come have a look &mdash; and any feedback is welcome!</p> +<!-- more--> + +<h2 id="gradual-type-precision-as-retraction">Gradual Type Precision as Retraction</h2> + +<p><a href="http://maxsnew.github.io/docs/precision-as-retraction.pdf">Gradual Type Precision as Retraction</a> + <br />Max New + <br />2016</p> + +<blockquote> + <p>Gradually typed programming languages allow for a mix of precision of static type information, allowing advanced type features to be added to existing languages, while still supporting interoperability with legacy code. The advantages of gradual typing are enticing to researchers and practitioners alike, but a general theory of gradually typed languages is only beginning to emerge after a decade of research.</p> + <p>It has long been noted that there is much similarity between work on contracts and gradual typing, and the use of retracts in domain theory which were used to relate models of untyped and typed lambda calculus in <a href="https://pdfs.semanticscholar.org/359e/ca57fe42d97cbb67f0b5591869abe5eb5421.pdf">Scott(1976)</a> and <a href="http://andrewkish-name.s3.amazonaws.com/scott80.pdf">Scott(1980)</a>. Here we take this connection seriously and consider how judgments in modern gradually typed languages can be framed in terms of retractions. While retractions in programming languages were originally studied in terms of denotational semantics in domains, our presentation will use only the most basic elements of category theory: composition, identity and equality of terms, so our formulation is equally applicable to axiomatic or operational semantics.</p> + <p>In particular we propose a semantic criterion for the notion of precision of gradual types, a common judgment in gradually typed languages (sometimes called naive subtyping for historical reasons). We relate it to a previous definition from <a href="https://www.eecs.northwestern.edu/%7Erobby/pubs/papers/esop2009-wf.pdf">Wadler and Findler(2009)</a> that defines type precision in terms of blame. We show that our definition decomposes in a similar way into “positive” and “negative” type precision, but without depending on a specific notion of blame in the language.</p></blockquote> + +<h2 id="linking-types-specifying-safe-interoperability-and-equivalences">Linking Types: Specifying Safe Interoperability and Equivalences</h2> + +<p><a href="https://dbp.io/pubs/2016/linking-types-poplsrc2017-proposal.pdf">Linking Types: Specifying Safe Interoperability and Equivalences</a> + <br />Daniel Patterson + <br />2016</p> + +<blockquote> + <p>All programs written in high-level languages link with libraries written in lower-level languages, often to expose constructs, like threads, random numbers, or automatic serialization, that aren’t possible in the high-level language. This linking usually takes place after compiling both languages to a common language, possibly assembly. In this sense, reasoning about crosslanguage linking means reasoning about compilation.</p> + <p>While most languages include cross-language linking (FFI) mechanisms, they are ad-hoc and can easily break the semantic equivalences of the source language, making it hard for source programmers to reason about correctness of their programs and hard for compiler writers to reason about correctness of their optimizations.</p> + <p>In this work, I design and motivate linking types, a language-based mechanism for formally specifying safe linking with libraries utilizing features inexpressible in the source. Linking types allows programmers to reason about their programs in the presence of behavior inexpressible in their language, without dealing with the intricacies of either the compiler or the particular language they are linking with.</p></blockquote> + +<h2 id="pruning-contracts-with-rosette">Pruning Contracts with Rosette</h2> + +<p><a href="http://www.ccs.neu.edu/home/types/resources/popl2017-src.pdf">Pruning Contracts with Rosette</a> + <br />Ben Greenman + <br />2016</p> + +<blockquote> + <p><a href="http://www.ccs.neu.edu/racket/pubs/icfp16-dnff.pdf">Contracts</a> are a pragmatic tool for managing software systems, but programs using contracts suffer runtime overhead. If this overhead becomes a performance bottleneck, programmers must manually edit or remove their contracts. This is no good. Rather, the contracts should identify their own inefficiencies and remove unnecessary dynamic checks. Implementing contracts with <a href="https://emina.github.io/rosette/">Rosette</a> is a promising way to build such self-aware contracts.</p></blockquote> + +<h2 id="while-were-at-it-lets-rant-on-srcs">While we&rsquo;re at it let&rsquo;s rant on SRCs</h2> + +<p>These abstracts are submitted at POPL&rsquo;s &ldquo;Student Research Competition&rdquo;. You submit an abstract, and if you get accepted to that thing, you get a bit of travel support money, and you have to prepare a poster and present it at the conference.</p> + +<p>I have a firm dislike for the <em>Competition</em> part of that concept: I think that people think of research too competitively already, and that we should have less of that, not more. (Having some is unfortunately unavoidable in scarce-resource situations.) I think that the process of awarding prizes to students with the &ldquo;best poster&rdquo; is dumb &mdash; and borderline ridiculous.</p> + +<p>On the other hand, my experience seeing them writing these extended abstracts is that it&rsquo;s a useful exercise for them, and produces nice result &mdash; short, readable introductions to their ideas. And Jennifer Paykin <a href="https://github.com/gasche/icfp2016-blog/blob/master/SVs/jennifer_paykin.md">convincingly argues</a> that although writing a poster is rather painful, actually presenting it during the conference is interesting and useful. In her words, &ldquo;it&rsquo;s worth it to get the experience of authentic and fruitful discussions&rdquo;. Plus having posters in the corridors of one&rsquo;s lab is very nice.</p> + +<p>I think we could have &ldquo;Student Research Sessions&rdquo; or &ldquo;Student Poster Sessions&rdquo;, where students are encouraged to present their work, would write those nice extended abstracts and posters, interact with researchers at the conference, and get travel money, without the ranking and prize stuff. (I would still encourage students to participate to SRC today, it seems to be worth it.)</p> + + Emacs daemon for fast editor startup + http://prl.ccs.neu.edu/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-17-emacs-daemon-for-fast-editor-startup + Mon, 17 Oct 2016 21:48:25 UT + Gabriel Scherer + +<p>In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs&rsquo; bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors &mdash; on each git commit for example, or when I use a <a href="https://addons.mozilla.org/en-US/firefox/addon/its-all-text/">Firefox extension</a> to edit a textarea content in a proper editor.</p> + +<p>In this blog post, I describe how to use <code>emacsclient</code> to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.</p> +<!-- more--> + +<p>Emacs has long supported a client/server mode: a daemon emacs instance is loaded in the background, and whenever you request a new emacs window you can creates a new frame (~ window) using the same instance. This means that the startup time is dramatically reduced after the daemon is launched, as for example the execution of your personal configuration code does not have to be repeated.</p> + +<p>To use it, I have this code as <code>/usr/bin/editor</code>:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="ch">#!/bin/bash</span> +emacsclient -a <span class="s2">""</span> -c <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The empty <code>-a</code> parameter means that if no daemon exists, it should start one in the background and retry. The <code>-c</code> option means that a new frame (window) should be created instead of reusing an existing one. <code>"$@"</code>means that when the script is invoked with a path as command-line parameter (<code>editor /tmp/foo.txt</code>), the corresponding file will be opened.</p> + +<p>Finally, my <code>.bash_profile</code> sets the <code>EDITOR</code> variable to <code>editor</code> (<code>export EDITOR=/usr/bin/editor</code>); this environment variable is what most tools (git included) will use to invoke a text editor.</p> + +<p>On my machine, starting the daemon takes 1.4s. Creating a client windows takes around 0.150s.</p> + +<p>If you want to control the environment in which the daemon process is started, you can launch it explicitly by running <code>emacs --daemon</code>.</p> + +<p>Cool kids use <a href="http://spacemacs.org/">Spacemacs</a> these days, which comes with all sort of convenient settings built in, and I&rsquo;m told that it does daemonization out of the box. I haven&rsquo;t taken the time to give Spacemacs a try yet.</p> + +<p>Finally, sometimes having all editor windows share the same process is not the right thing, because I do stuff which makes Emacs a bit unstable. (It&rsquo;s not very good at asynchronous communication with the rest of the world, so for example accessing a file through SSH from Emacs can hang the process when network goes bad.). I&rsquo;ve been bitten a few times by a crash of all editor windows at the same time, and since then, when I know I&rsquo;m going to do &ldquo;heavy stuff&rdquo;, I launch a separate process for it (just <code>emacs</code> instead of <code>editor</code> or <code>emacsclient</code>).</p> + +<p>P.S.: To precisely measure the startup time, ask Emacs to evaluate a Lisp expression on startup, to kill it immediately.:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span>$ <span class="nb">time</span> emacs --eval <span class="s2">"(save-buffers-kill-terminal)"</span> +$ <span class="nb">time</span> emacsclient -a <span class="s1">&#39;&#39;</span> -c -e <span class="s2">"(save-buffers-kill-terminal)"</span> +</pre></div> +</td></tr></tbody></table> +</div> + + Does anyone still care about printed proceedings? (Grab some at NEU this week!) + http://prl.ccs.neu.edu/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-13-does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week + Mon, 13 Jun 2016 10:50:14 UT + Gabriel Scherer + +<p>Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!</p> +<!-- more--> + +<p>If you are in the area and are interested, feel free to send me an email and come grab them. We have ASPLOS from XIII to XX, PLDI from 2005 to 2015 (but not 2014), and OOPSLA from 2002 to 2015. When I saw the stack, I grabbed the POPL ones from 2002 to 2015, but in fact I have no idea what to do with them and I&rsquo;m a bit skeptical they would be of use to me; if you know you would use them, I would be glad to let you have them.</p> + +<p>If you were to buy those proceedings at conference-subscription rates today, it would cost you a small fortune. Yet nobody seems to want them. An odd disconnect, that I found amusing and maybe worthy of a blog post.</p> + +<p>But don&rsquo;t get me wrong, the future of printed proceedings is not an important question. We should rather be asking ourselves: why are the products of the work of our communities not easily accessible in an Open Access long-term archive? Are you submitting your articles to arXiv, or another archive? Why not?</p> + +<p>Not caring about printed proceedings is perfectly fine; but please care about people outside institutions that want to access your work &mdash; for example, master student myself.</p> + +<hr /> + +<p><em>Update (August 2017):</em> Gabriel returned to France and left the POPL proceedings at his desk. The proceedings are sitting in the hallway at NEU, in case anyone cares.</p> + + ICFP 2016: looking for student volunteers + http://prl.ccs.neu.edu/blog/2016/06/07/icfp-2016-looking-for-student-volunteers/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-07-icfp-2016-looking-for-student-volunteers + Tue, 07 Jun 2016 11:53:47 UT + Gabriel Scherer + +<p>If you are a student, you should consider <a href="http://goo.gl/forms/fKg3vpjNryBlGWB32">applying</a> to become an ICFP 2016 student volunteer! The deadline for application is July 31st, 2016.</p> +<!-- more--> + +<p><a href="http://conf.researchr.org/attending/icfp-2016/Student+Volunteers">ICFP 2016</a>, the Internal Conference on Functional Programming, is happening in Nara, Japan. If you are a student, you may be interest in being a Student Volunteer: you help run the conference, and in exchange do not pay registration fees &mdash; but you still have to find funding for the travel, hosting, and dinners. Quoting the <a href="http://conf.researchr.org/attending/icfp-2016/Student+Volunteers">Student Volunteer</a> webpage:</p> + +<blockquote> + <p>ICFP is pleased to offer a number of opportunities for student volunteers, who are vital to the efficient operation and continued success of the conference each year. The student volunteer program is a chance for students from around the world to participate in the conferences whilst assisting us in preparing and running the event.</p> + <p>Job assignments for student volunteers include assisting with technical sessions, workshops, tutorials and panels, helping the registration desk, operating the information desk, helping with traffic flow, and general assistance to keep the conferences running smoothly.</p> + <p>The Student Volunteer Program helps more students attend the ICFP conference by covering conference fees (but not travel or lodging expenses) in exchange for a fixed number of work hours (usually from 8 to 12) helping with the conference organization (registration and information desks, assistance during talk sessions, etc.).</p> + <p>The Student Volunteer registration covers:</p> + <ul> + <li>Access to all workshops and the main conference,</li> + <li>Daily lunches and coffee breaks,</li> + <li>Access to social events, including the banquet.</li></ul> + <p>To apply, please fill the <a href="http://goo.gl/forms/fKg3vpjNryBlGWB32">following form</a>.</p> + <p>The application deadline is July 31st, 2016. Applications after this date may be considered pending availability.</p> + <p>You can send questions about student volunteering to <code>icfp-SV at researchr dot org</code>.</p></blockquote> + +<p>I was &ldquo;student volunteer captain&rdquo; at ICFP last year in Vancouver, and I will do it again this year. My entirely personal take on the thing is that being a Student Volunteer is worth it, but that being a Captain is too much work.</p> + +<p>The main downside of being a volunteer is some of the shifts are at the registration desk, and they may imply missing some of the talks &mdash; and also you may have to get up early for your duties. The upsides are many. You get belong to a small group of nice people. You have interactions with many people without much effort; you will enjoy the sparks of gratitude in the eyes of the &ldquo;Where is Workshop Room B2?&rdquo; crowd. You have a small peek into the kind of work involved in running a conference; most people actually working on the organization (we SVs are hobbyists) are pouring surprising amount of work. Also, you learn to fold tee-shirts very well, if you&rsquo;re on &ldquo;bag stuffing&rdquo; duty.</p> + +<p>Being a student volunteer can be combined with other forms of travel support, such as SIGPLAN PAC funding; see the <a href="http://conf.researchr.org/attending/icfp-2016/student-travel-support">travel support page</a> for more details.</p> + +<p>Another thing you should think about is applying to <a href="http://conf.researchr.org/track/icfp-2016/PLMW-ICFP-2016">PLMW</a>, the Programming Languages Mentoring Workshop that happens at ICFP, POPL, and PLDI. PLMW funding covers the whole conference cost (travel, housing, registration, dinners), so if you get PLMW funding you have no financial motivation to be a student volunteer. This year, PLMW focuses on early career graduate students.</p> + + Measuring GC latencies in Haskell, OCaml, Racket + http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + Tue, 24 May 2016 10:51:34 UT + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> + + NEPLS on May 31st at UMass, Amherst + http://prl.ccs.neu.edu/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/?utm_source=Author-Gabriel-Scherer&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-03-nepls-on-may-31st-at-umass-amherst + Tue, 03 May 2016 08:21:07 UT + Gabriel Scherer + +<p>It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!</p> +<!-- more--> + +<blockquote> + <p>The next New England Programming Languages and Systems Symposium will take place on Tuesday, May 31st 2016 at University of Massachusetts, Amherst. Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is Tuesday, May 17th. Send your proposal to talks@nepls.org.</p> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <p> http://www.nepls.org/</p></blockquote> + +<p>I&rsquo;m personally fond of such regional events, which is a great time to learn about the research around us in a less formal and exhausting setting than a 200-attendees conference.</p> + +<p>If you are in the area, please consider applying to talk about your work. If one of your colleague is working on something you find exciting, please invite them to apply!</p> \ No newline at end of file diff --git a/blog/feeds/Author-Jan-Vitek.atom.xml b/blog/feeds/Author-Jan-Vitek.atom.xml new file mode 100644 index 00000000..4874def0 --- /dev/null +++ b/blog/feeds/Author-Jan-Vitek.atom.xml @@ -0,0 +1,45 @@ + + + PRL Blog: Posts tagged 'Author: Jan Vitek' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Jan-Vitek-html + 2017-02-28T23:01:00Z + + PLISS: Oregon without the greek + + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-pliss-oregon-without-the-greek + 2017-02-28T23:01:00Z + 2017-02-28T23:01:00Z + + Jan Vitek + +<p>What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or <a href="https://pliss2017.github.io">PLISS</a> for short) has set out to answer.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2017.github.io/speakers.html">twelve speakers</a> versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.</p> + +<p>Lectures cover current research and future trends in programming language design and implementation, including:</p> + +<ul> + <li>Writing Just-in-time Compilers with LLVM with Jan Vitek</li> + <li>Performance Evaluation and Benchmarking with Laurie Tratt</li> + <li>Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller</li> + <li>High-Performance Fully Concurrent Garbage Collection with Richard Jones</li> + <li>Compiling Dynamic Languages with David Edelsohn</li> + <li>Language-support for Distributed Datastores with Suresh Jagannathan</li> + <li>Testing Programming Language Implementations with Alastair Donaldson</li> + <li>Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might</li> + <li>The evolution of Scala with Martin Odersky</li></ul> + +<p>A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.</p> + +<p>Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.</p> + +<p>More information at:</p> + +<p><a href="https://pliss2017.github.io">https://pliss2017.github.io</a></p> + +<p>(Oregon is a reference to the <a href="https://www.cs.uoregon.edu/research/summerschool/summer17/">OPLSS</a>, in which you may also be interested.)</p> \ No newline at end of file diff --git a/blog/feeds/Author-Jan-Vitek.rss.xml b/blog/feeds/Author-Jan-Vitek.rss.xml new file mode 100644 index 00000000..10800430 --- /dev/null +++ b/blog/feeds/Author-Jan-Vitek.rss.xml @@ -0,0 +1,45 @@ + + + + PRL Blog: Posts tagged 'Author: Jan Vitek' + PRL Blog: Posts tagged 'Author: Jan Vitek' + http://prl.ccs.neu.edu/blog/tags/Author-Jan-Vitek.html + Tue, 28 Feb 2017 23:01:00 UT + Tue, 28 Feb 2017 23:01:00 UT + 1800 + + PLISS: Oregon without the greek + http://prl.ccs.neu.edu/blog/2017/02/28/pliss-oregon-without-the-greek/?utm_source=Author-Jan-Vitek&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-pliss-oregon-without-the-greek + Tue, 28 Feb 2017 23:01:00 UT + Jan Vitek + +<p>What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or <a href="https://pliss2017.github.io">PLISS</a> for short) has set out to answer.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2017.github.io/speakers.html">twelve speakers</a> versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.</p> + +<p>Lectures cover current research and future trends in programming language design and implementation, including:</p> + +<ul> + <li>Writing Just-in-time Compilers with LLVM with Jan Vitek</li> + <li>Performance Evaluation and Benchmarking with Laurie Tratt</li> + <li>Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller</li> + <li>High-Performance Fully Concurrent Garbage Collection with Richard Jones</li> + <li>Compiling Dynamic Languages with David Edelsohn</li> + <li>Language-support for Distributed Datastores with Suresh Jagannathan</li> + <li>Testing Programming Language Implementations with Alastair Donaldson</li> + <li>Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might</li> + <li>The evolution of Scala with Martin Odersky</li></ul> + +<p>A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.</p> + +<p>Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.</p> + +<p>More information at:</p> + +<p><a href="https://pliss2017.github.io">https://pliss2017.github.io</a></p> + +<p>(Oregon is a reference to the <a href="https://www.cs.uoregon.edu/research/summerschool/summer17/">OPLSS</a>, in which you may also be interested.)</p> \ No newline at end of file diff --git a/blog/feeds/Author-Jonathan-Schuster.atom.xml b/blog/feeds/Author-Jonathan-Schuster.atom.xml new file mode 100644 index 00000000..adbe172f --- /dev/null +++ b/blog/feeds/Author-Jonathan-Schuster.atom.xml @@ -0,0 +1,16 @@ + + + PRL Blog: Posts tagged 'Author: Jonathan Schuster' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Jonathan-Schuster-html + 2016-11-30T15:24:45Z + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-30-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages + 2016-11-30T15:24:45Z + 2016-11-30T15:24:45Z + + Jonathan Schuster + \ No newline at end of file diff --git a/blog/feeds/Author-Jonathan-Schuster.rss.xml b/blog/feeds/Author-Jonathan-Schuster.rss.xml new file mode 100644 index 00000000..06c9868c --- /dev/null +++ b/blog/feeds/Author-Jonathan-Schuster.rss.xml @@ -0,0 +1,16 @@ + + + + PRL Blog: Posts tagged 'Author: Jonathan Schuster' + PRL Blog: Posts tagged 'Author: Jonathan Schuster' + http://prl.ccs.neu.edu/blog/tags/Author-Jonathan-Schuster.html + Wed, 30 Nov 2016 15:24:45 UT + Wed, 30 Nov 2016 15:24:45 UT + 1800 + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + http://prl.ccs.neu.edu/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/?utm_source=Author-Jonathan-Schuster&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-30-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages + Wed, 30 Nov 2016 15:24:45 UT + Jonathan Schuster + \ No newline at end of file diff --git a/blog/feeds/Author-Julia-Belyakova.atom.xml b/blog/feeds/Author-Julia-Belyakova.atom.xml new file mode 100644 index 00000000..fdbad088 --- /dev/null +++ b/blog/feeds/Author-Julia-Belyakova.atom.xml @@ -0,0 +1,84 @@ + + + PRL Blog: Posts tagged 'Author: Julia Belyakova' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Julia-Belyakova-html + 2017-06-16T11:38:25Z + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Julia-Belyakova.rss.xml b/blog/feeds/Author-Julia-Belyakova.rss.xml new file mode 100644 index 00000000..b58388a4 --- /dev/null +++ b/blog/feeds/Author-Julia-Belyakova.rss.xml @@ -0,0 +1,76 @@ + + + + PRL Blog: Posts tagged 'Author: Julia Belyakova' + PRL Blog: Posts tagged 'Author: Julia Belyakova' + http://prl.ccs.neu.edu/blog/tags/Author-Julia-Belyakova.html + Fri, 16 Jun 2017 11:38:25 UT + Fri, 16 Jun 2017 11:38:25 UT + 1800 + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=Author-Julia-Belyakova&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Justin-Slepak.atom.xml b/blog/feeds/Author-Justin-Slepak.atom.xml new file mode 100644 index 00000000..8cad1864 --- /dev/null +++ b/blog/feeds/Author-Justin-Slepak.atom.xml @@ -0,0 +1,54 @@ + + + PRL Blog: Posts tagged 'Author: Justin Slepak' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Justin-Slepak-html + 2017-05-04T18:26:48Z + + Rank Polymorphism + + urn:http-prl-ccs-neu-edu:-blog-2017-05-04-rank-polymorphism + 2017-05-04T18:26:48Z + 2017-05-04T18:26:48Z + + Justin Slepak + +<p>Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let&rsquo;s just call it <code>lerp</code>) for scalars:</p> + +<pre><code>(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))</code></pre> + +<p>The number marks on each argument indicate the expected &ldquo;rank&rdquo; of the argument: how many dimensions it should have. In this case, each one is marked <code>0</code>, indicating a scalar (<em>i.e.</em>, 0-dimensional) argument. The function is usable as-is for</p> + +<ul> + <li> + <p>α-blending two RGB pixels</p></li> + <li> + <p>dimming or brightening an image</p></li> + <li> + <p>fade transition between video scenes</p></li></ul> +<!-- more--> + +<p>Each of these use cases mixes the argument dimensions a little differently. A pixel is a vector (a rank&ndash;1 structure) of numbers representing color channel values, so the α-blending case uses two vector arguments and one scalar argument.</p> + +<p>The only real difference between these use cases is the iteration space: they&rsquo;re all effectively loop nests around the same basic scalar operation. In a rank-polymorphic language, the iteration space is derived automatically from the data, so you don&rsquo;t need to write out the control structure yourself.</p> + +<p>The fundamental idea behind function application here is breaking the argument arrays into lower-ranked pieces called &ldquo;cells.&rdquo; Each cell has the rank expected by the function being applied. In the case of <code>lerp</code>, the pixels, images, videos, etc. are all broken up into rank&ndash;0 (scalar) cells because <code>lerp</code> expects rank&ndash;0 arguments. Other expected ranks are possible as well— a vector dot product function <code>dot-prod</code> would call for rank&ndash;1 cells, and a matrix inversion function <code>minv</code> would call for rank&ndash;2 cells.</p> + +<p>The structure built up around the cells is called the &ldquo;frame.&rdquo; A matrix array is a rank&ndash;2 frame containing rank&ndash;0 cells for <code>lerp</code>, but it would be a rank&ndash;1 frame containing rank&ndash;1 cells for <code>dot-prod</code> and a rank&ndash;0 frame containing a single rank&ndash;1 cell for <code>minv</code>. A rank-<em>n</em> array could be broken down into a frame of cells in <em>n+1</em> different ways, and it&rsquo;s the function being applied that determines which decomposition to use.</p> + +<p>Unfortunately, the implicit control structure that&rsquo;s so convenient for the programmer is a problem for a compiler. Historically, implementations of such languages have had to do without static information about the iteration space. Interpreters (and line-at-a-time compilers, to a lesser extent) get to inspect the concrete data they&rsquo;re dealing with, but static compilers have had to make do with emitting a generic loop structure. A &ldquo;loop&rdquo; over a scalar might sound like trivial overhead, but not when it appears within some other hot loop. Being unable to see when loop boundaries match up is also a barrier to loop fusion. The lack of thorough static shape information was a long-standing problem my advisor pointed out to me when I was a new student looking at possible research projects, and he was interested in applying some form of static analysis to gather that information.</p> + +<p>The first step in addressing it was to come up with a formal semantics for rank polymorphism. Although <a href="http://www.jsoftware.com/papers/APL.htm">APL has existed since the 1960s</a>, it had mostly lived in a separate world from mainstream programming language research. The formal techniques developed in PL had seen little to no application to APL and its &ldquo;successor&rdquo; language J.</p> + +<p>There&rsquo;s a lot to dislike about APL and J—special case behavior in many of the primitive operators, limited function arity, syntactic separation of first-order and second-order functions, the impossibility of parsing an entire program at once (fun fact: static analysis <a href="http://dl.acm.org/citation.cfm?id=805380">has been tried</a> there)—and of course the idiosyncratic identifiers used for primops have prompted plenty of internet arguments. None of those things are essential to the programming model, so I&rsquo;m <a href="http://www.ccs.neu.edu/home/jrslepak/proposal.pdf">building a new language called Remora</a> to isolate the aspects I want to study.</p> + +<p>People don&rsquo;t always think of a type system as a form of static analysis, but it turned out to be an effective way of gathering shape information. Remora&rsquo;s <a href="http://www.ccs.neu.edu/home/jrslepak/esop14-full.pdf">type system</a> uses a restricted form of dependent types, in the style of <a href="https://www.cs.cmu.edu/~rwh/theses/xi.pdf">Dependent ML</a>. An array type is indexed by the shape, the numeric sizes of the array&rsquo;s individual dimensions. Index polymorphism (<em>i.e.</em>, Π types) allows functions to work on varying cell shapes and even varying cell ranks (which is essential for primitives like <code>append</code> and <code>reduce</code>, which operate along the major axis of arrays, no matter their rank). Frame-rank polymorphism, which gives rise to the control structure, remains completely implicit, leaving it to be identified by the type rule for function application. As a nice bonus, type soundness rules out run-time errors arising from incompatible argument shapes.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2016/10/19/history-of-actors/">History of Actors</a></li> + <li><a href="http://prl.ccs.neu.edu/blog/2017/02/21/datalog-for-static-analysis/">Datalog for Static Analysis</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Justin-Slepak.rss.xml b/blog/feeds/Author-Justin-Slepak.rss.xml new file mode 100644 index 00000000..828634c8 --- /dev/null +++ b/blog/feeds/Author-Justin-Slepak.rss.xml @@ -0,0 +1,54 @@ + + + + PRL Blog: Posts tagged 'Author: Justin Slepak' + PRL Blog: Posts tagged 'Author: Justin Slepak' + http://prl.ccs.neu.edu/blog/tags/Author-Justin-Slepak.html + Thu, 04 May 2017 18:26:48 UT + Thu, 04 May 2017 18:26:48 UT + 1800 + + Rank Polymorphism + http://prl.ccs.neu.edu/blog/2017/05/04/rank-polymorphism/?utm_source=Author-Justin-Slepak&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-04-rank-polymorphism + Thu, 04 May 2017 18:26:48 UT + Justin Slepak + +<p>Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let&rsquo;s just call it <code>lerp</code>) for scalars:</p> + +<pre><code>(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))</code></pre> + +<p>The number marks on each argument indicate the expected &ldquo;rank&rdquo; of the argument: how many dimensions it should have. In this case, each one is marked <code>0</code>, indicating a scalar (<em>i.e.</em>, 0-dimensional) argument. The function is usable as-is for</p> + +<ul> + <li> + <p>α-blending two RGB pixels</p></li> + <li> + <p>dimming or brightening an image</p></li> + <li> + <p>fade transition between video scenes</p></li></ul> +<!-- more--> + +<p>Each of these use cases mixes the argument dimensions a little differently. A pixel is a vector (a rank&ndash;1 structure) of numbers representing color channel values, so the α-blending case uses two vector arguments and one scalar argument.</p> + +<p>The only real difference between these use cases is the iteration space: they&rsquo;re all effectively loop nests around the same basic scalar operation. In a rank-polymorphic language, the iteration space is derived automatically from the data, so you don&rsquo;t need to write out the control structure yourself.</p> + +<p>The fundamental idea behind function application here is breaking the argument arrays into lower-ranked pieces called &ldquo;cells.&rdquo; Each cell has the rank expected by the function being applied. In the case of <code>lerp</code>, the pixels, images, videos, etc. are all broken up into rank&ndash;0 (scalar) cells because <code>lerp</code> expects rank&ndash;0 arguments. Other expected ranks are possible as well— a vector dot product function <code>dot-prod</code> would call for rank&ndash;1 cells, and a matrix inversion function <code>minv</code> would call for rank&ndash;2 cells.</p> + +<p>The structure built up around the cells is called the &ldquo;frame.&rdquo; A matrix array is a rank&ndash;2 frame containing rank&ndash;0 cells for <code>lerp</code>, but it would be a rank&ndash;1 frame containing rank&ndash;1 cells for <code>dot-prod</code> and a rank&ndash;0 frame containing a single rank&ndash;1 cell for <code>minv</code>. A rank-<em>n</em> array could be broken down into a frame of cells in <em>n+1</em> different ways, and it&rsquo;s the function being applied that determines which decomposition to use.</p> + +<p>Unfortunately, the implicit control structure that&rsquo;s so convenient for the programmer is a problem for a compiler. Historically, implementations of such languages have had to do without static information about the iteration space. Interpreters (and line-at-a-time compilers, to a lesser extent) get to inspect the concrete data they&rsquo;re dealing with, but static compilers have had to make do with emitting a generic loop structure. A &ldquo;loop&rdquo; over a scalar might sound like trivial overhead, but not when it appears within some other hot loop. Being unable to see when loop boundaries match up is also a barrier to loop fusion. The lack of thorough static shape information was a long-standing problem my advisor pointed out to me when I was a new student looking at possible research projects, and he was interested in applying some form of static analysis to gather that information.</p> + +<p>The first step in addressing it was to come up with a formal semantics for rank polymorphism. Although <a href="http://www.jsoftware.com/papers/APL.htm">APL has existed since the 1960s</a>, it had mostly lived in a separate world from mainstream programming language research. The formal techniques developed in PL had seen little to no application to APL and its &ldquo;successor&rdquo; language J.</p> + +<p>There&rsquo;s a lot to dislike about APL and J—special case behavior in many of the primitive operators, limited function arity, syntactic separation of first-order and second-order functions, the impossibility of parsing an entire program at once (fun fact: static analysis <a href="http://dl.acm.org/citation.cfm?id=805380">has been tried</a> there)—and of course the idiosyncratic identifiers used for primops have prompted plenty of internet arguments. None of those things are essential to the programming model, so I&rsquo;m <a href="http://www.ccs.neu.edu/home/jrslepak/proposal.pdf">building a new language called Remora</a> to isolate the aspects I want to study.</p> + +<p>People don&rsquo;t always think of a type system as a form of static analysis, but it turned out to be an effective way of gathering shape information. Remora&rsquo;s <a href="http://www.ccs.neu.edu/home/jrslepak/esop14-full.pdf">type system</a> uses a restricted form of dependent types, in the style of <a href="https://www.cs.cmu.edu/~rwh/theses/xi.pdf">Dependent ML</a>. An array type is indexed by the shape, the numeric sizes of the array&rsquo;s individual dimensions. Index polymorphism (<em>i.e.</em>, Π types) allows functions to work on varying cell shapes and even varying cell ranks (which is essential for primitives like <code>append</code> and <code>reduce</code>, which operate along the major axis of arrays, no matter their rank). Frame-rank polymorphism, which gives rise to the control structure, remains completely implicit, leaving it to be identified by the type rule for function application. As a nice bonus, type soundness rules out run-time errors arising from incompatible argument shapes.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2016/10/19/history-of-actors/">History of Actors</a></li> + <li><a href="http://prl.ccs.neu.edu/blog/2017/02/21/datalog-for-static-analysis/">Datalog for Static Analysis</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Kevin-Clancy.atom.xml b/blog/feeds/Author-Kevin-Clancy.atom.xml new file mode 100644 index 00000000..a95ec473 --- /dev/null +++ b/blog/feeds/Author-Kevin-Clancy.atom.xml @@ -0,0 +1,168 @@ + + + PRL Blog: Posts tagged 'Author: Kevin Clancy' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Kevin-Clancy-html + 2017-10-22T11:59:06Z + + Monotonicity Types: Towards A Type System for Eventual Consistency + + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + 2017-10-22T11:59:06Z + 2017-10-22T11:59:06Z + + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> + + Refinement Types + + urn:http-prl-ccs-neu-edu:-blog-2017-04-20-refinement-types + 2017-04-20T23:38:23Z + 2017-04-20T23:38:23Z + + Kevin Clancy + <!-- more--> + +<p>Roughly, a refinement type system is an extra layer of precision, enforced through subtyping, added onto an existing type system. A base type is decomposed into a set of <em>base refinements</em>, each of which denotes a subset of the values belonging to the base type. A subtyping relation respecting set inclusion can then be defined among the refinements of the base type. These subtyping relations can be lifted onto a subtyping relation for compound types using a standard arrow subtyping rule.</p> + +<p>Extra type-checking precision sounds great, but what in practical terms does this precision look like? Freeman and Pfenning&rsquo;s &rsquo;92 paper <em>Refinement Types for ML</em> proposes extending ML&rsquo;s type definition language with constructs for decomposing a discriminated union type into a lattice of subtypes. For example, it allows the decomposition of a list type into a lattice including base refinements for empty lists, non-empty lists, and singletons. Those with experience in functional programming will realize this alleviates the dreaded and inescapable “non-exhaustive pattern match” warning, which tends to crop up in situations where the programmer understands that an exhaustive pattern match is not necessary.</p> + +<p>In the late 90&rsquo;s Xi and Pfenning advanced the state of refinement types by introducing a dependent refinement type system, implemented as a tool called Dependent ML. Their approach identifies a base refinement using a tuple of terms drawn from some computationally tractable constraint language called an <em>index language</em>. A list datatype can then be refined with a term of the <em>linear integer arithmetic</em> index language, denoting the subset of all lists having a specific length. One list refinement is then considered a subtype of another when a constraint solver can prove their index terms equal. Vazou et. al.&rsquo;s recent project Liquid Haskell is another dependent refinement type system which decides subtyping among base types by invoking an SMT solver under a context-dependent set of constraints. It differs significantly from Dependent ML in that it refines base types with certain well-behaved program terms rather than indices.</p> + +<hr /> + +<p>Resources:</p> + +<ul> + <li><a href="/blog/static/refinement_types_lecture.pdf">Full Notes</a></li> + <li><a href="/blog/static/refinement_types_bib.pdf">Annotated Bibliography</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/refinement-types">GitHub</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Kevin-Clancy.rss.xml b/blog/feeds/Author-Kevin-Clancy.rss.xml new file mode 100644 index 00000000..8f4f5399 --- /dev/null +++ b/blog/feeds/Author-Kevin-Clancy.rss.xml @@ -0,0 +1,166 @@ + + + + PRL Blog: Posts tagged 'Author: Kevin Clancy' + PRL Blog: Posts tagged 'Author: Kevin Clancy' + http://prl.ccs.neu.edu/blog/tags/Author-Kevin-Clancy.html + Sun, 22 Oct 2017 11:59:06 UT + Sun, 22 Oct 2017 11:59:06 UT + 1800 + + Monotonicity Types: Towards A Type System for Eventual Consistency + http://prl.ccs.neu.edu/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/?utm_source=Author-Kevin-Clancy&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + Sun, 22 Oct 2017 11:59:06 UT + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> + + Refinement Types + http://prl.ccs.neu.edu/blog/2017/04/20/refinement-types/?utm_source=Author-Kevin-Clancy&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-04-20-refinement-types + Thu, 20 Apr 2017 23:38:23 UT + Kevin Clancy + <!-- more--> + +<p>Roughly, a refinement type system is an extra layer of precision, enforced through subtyping, added onto an existing type system. A base type is decomposed into a set of <em>base refinements</em>, each of which denotes a subset of the values belonging to the base type. A subtyping relation respecting set inclusion can then be defined among the refinements of the base type. These subtyping relations can be lifted onto a subtyping relation for compound types using a standard arrow subtyping rule.</p> + +<p>Extra type-checking precision sounds great, but what in practical terms does this precision look like? Freeman and Pfenning&rsquo;s &rsquo;92 paper <em>Refinement Types for ML</em> proposes extending ML&rsquo;s type definition language with constructs for decomposing a discriminated union type into a lattice of subtypes. For example, it allows the decomposition of a list type into a lattice including base refinements for empty lists, non-empty lists, and singletons. Those with experience in functional programming will realize this alleviates the dreaded and inescapable “non-exhaustive pattern match” warning, which tends to crop up in situations where the programmer understands that an exhaustive pattern match is not necessary.</p> + +<p>In the late 90&rsquo;s Xi and Pfenning advanced the state of refinement types by introducing a dependent refinement type system, implemented as a tool called Dependent ML. Their approach identifies a base refinement using a tuple of terms drawn from some computationally tractable constraint language called an <em>index language</em>. A list datatype can then be refined with a term of the <em>linear integer arithmetic</em> index language, denoting the subset of all lists having a specific length. One list refinement is then considered a subtype of another when a constraint solver can prove their index terms equal. Vazou et. al.&rsquo;s recent project Liquid Haskell is another dependent refinement type system which decides subtyping among base types by invoking an SMT solver under a context-dependent set of constraints. It differs significantly from Dependent ML in that it refines base types with certain well-behaved program terms rather than indices.</p> + +<hr /> + +<p>Resources:</p> + +<ul> + <li><a href="/blog/static/refinement_types_lecture.pdf">Full Notes</a></li> + <li><a href="/blog/static/refinement_types_bib.pdf">Annotated Bibliography</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/refinement-types">GitHub</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Leif-Andersen.atom.xml b/blog/feeds/Author-Leif-Andersen.atom.xml new file mode 100644 index 00000000..0f5bbf03 --- /dev/null +++ b/blog/feeds/Author-Leif-Andersen.atom.xml @@ -0,0 +1,325 @@ + + + PRL Blog: Posts tagged 'Author: Leif Andersen' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Leif-Andersen-html + 2022-01-06T17:56:08Z + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + 2022-01-06T17:56:08Z + 2022-01-06T17:56:08Z + + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> + + [Making an IDE Plugin for DrRacket (cross-post)](https://lang.video/blog/2018/03/21/making-an-ide-plugin-for-drracket/) + + urn:http-prl-ccs-neu-edu:-blog-2018-04-12-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket + 2018-04-12T12:12:53Z + 2018-04-12T12:12:53Z + + Leif Andersen + + + Racket 6.9 and Windows 10 Creators Update + + urn:http-prl-ccs-neu-edu:-blog-2017-05-26-racket-6-9-and-windows-10-creators-update + 2017-05-26T17:00:28Z + 2017-05-26T17:00:28Z + + Leif Andersen + +<p><a href="http://racket-lang.org/">Racket</a> 6.9 was released in April and it has been smooth sailing for many people. However, some people using the <a href="https://blogs.windows.com/windowsexperience/2017/04/11/whats-new-in-the-windows-10-creators-update/">Windows 10 Creators Update</a> have been experiencing <a href="https://github.com/racket/racket/issues/1671">crashes</a>, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.</p> + +<p>The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.</p> + +<p>A VirtualBox image with Racket preinstalled can be downloaded here:</p> + +<ul> + <li><a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova</a></li></ul> + +<p>The username and password for this machine are both <code>racket</code>.</p> +<!-- more--> + +<ol> + <li> + <p>The first thing you need to install is virtualization software. In principle it doesn&rsquo;t matter what you install, but for this tutorial, we will use <a href="https://www.virtualbox.org/">VirtualBox</a>. Go to their <a href="https://www.virtualbox.org/wiki/Downloads">downloads</a> page and download and install the version for your platform (most likely Windows).</p></li> + <li> + <p>Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which <a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">you can download here</a>. The rest of this tutorial will assume you are using this image.</p></li> + <li> + <p>Start up VirtualBox and import the virtual machine. You can do this by clicking on <code>File -&gt; Import Appliance</code>. In the dialog, select the image you downloaded and hit <code>Continue</code>. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click <code>Import</code>.</p></li> + <li> + <p>After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit <code>Start</code>. Once started up, you will see DrRacket and Firefox on the VM&rsquo;s desktop.</p></li></ol> + + Type-Directed Compilation, Parts I and II + + urn:http-prl-ccs-neu-edu:-blog-2017-04-17-type-directed-compilation-parts-i-and-ii + 2017-04-17T12:00:17Z + 2017-04-17T12:00:17Z + Leif Andersen + William J. Bowman + + Leif Andersen, William J. Bowman + <!-- more--> + +<h3 id="part-i-type-directed-compilation-by-leif-andersen">Part I: <em>Type-Directed Compilation</em>, by Leif Andersen.</h3> + +<p>In this talk we discuss the history of type directed compilation. We start with Xavier Leroy&rsquo;s seminal paper: <a href="http://gallium.inria.fr/~xleroy/publi/unboxed-polymorphism.pdf"><em>Unboxed Objects and Polymorphic Typing</em></a>, continue to <a href="https://www.cs.cmu.edu/~rwh/papers/til/pldi96.pdf">TIL</a> (Typed Intermediate Language), and finish up with <a href="https://dash.harvard.edu/handle/1/2797451">TAL</a> (Typed Assembly Language). We talk about what it means for a compiler to be typed preserving, and give examples of optimizations that are enabled by types.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-24.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;24.md</a></li></ul> + +<h3 id="part-ii-dependent-type-directed-compilation-by-william-j-bowman">Part II: <em>Dependent Type-Directed Compilation</em>, by William J. Bowman</h3> + +<p>A certifying compiler is not verified, but it produces a proof of correctness for each binary. This proof can be independently checked to show that the binary was compiled correctly, removing the compiler from the trusted code base. Certifying compilation has its roots in preserving type-preserving compilation, and in particular in preserving dependent types. We start the history of dependent-type-preserving compilation with a compiler from C to Assembly. We&rsquo;ll see a result showing that preserving dependent types isn&rsquo;t possible, and then we&rsquo;ll do it anyway.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-28.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;28.md</a></li></ul> + +<p>Notes (to appear here, eventually):</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation">https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Leif-Andersen.rss.xml b/blog/feeds/Author-Leif-Andersen.rss.xml new file mode 100644 index 00000000..0d9ce171 --- /dev/null +++ b/blog/feeds/Author-Leif-Andersen.rss.xml @@ -0,0 +1,317 @@ + + + + PRL Blog: Posts tagged 'Author: Leif Andersen' + PRL Blog: Posts tagged 'Author: Leif Andersen' + http://prl.ccs.neu.edu/blog/tags/Author-Leif-Andersen.html + Thu, 06 Jan 2022 17:56:08 UT + Thu, 06 Jan 2022 17:56:08 UT + 1800 + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + http://prl.ccs.neu.edu/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/?utm_source=Author-Leif-Andersen&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + Thu, 06 Jan 2022 17:56:08 UT + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> + + [Making an IDE Plugin for DrRacket (cross-post)](https://lang.video/blog/2018/03/21/making-an-ide-plugin-for-drracket/) + http://prl.ccs.neu.edu/blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/?utm_source=Author-Leif-Andersen&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-04-12-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket + Thu, 12 Apr 2018 12:12:53 UT + Leif Andersen + + + Racket 6.9 and Windows 10 Creators Update + http://prl.ccs.neu.edu/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/?utm_source=Author-Leif-Andersen&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-26-racket-6-9-and-windows-10-creators-update + Fri, 26 May 2017 17:00:28 UT + Leif Andersen + +<p><a href="http://racket-lang.org/">Racket</a> 6.9 was released in April and it has been smooth sailing for many people. However, some people using the <a href="https://blogs.windows.com/windowsexperience/2017/04/11/whats-new-in-the-windows-10-creators-update/">Windows 10 Creators Update</a> have been experiencing <a href="https://github.com/racket/racket/issues/1671">crashes</a>, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.</p> + +<p>The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.</p> + +<p>A VirtualBox image with Racket preinstalled can be downloaded here:</p> + +<ul> + <li><a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova</a></li></ul> + +<p>The username and password for this machine are both <code>racket</code>.</p> +<!-- more--> + +<ol> + <li> + <p>The first thing you need to install is virtualization software. In principle it doesn&rsquo;t matter what you install, but for this tutorial, we will use <a href="https://www.virtualbox.org/">VirtualBox</a>. Go to their <a href="https://www.virtualbox.org/wiki/Downloads">downloads</a> page and download and install the version for your platform (most likely Windows).</p></li> + <li> + <p>Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which <a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">you can download here</a>. The rest of this tutorial will assume you are using this image.</p></li> + <li> + <p>Start up VirtualBox and import the virtual machine. You can do this by clicking on <code>File -&gt; Import Appliance</code>. In the dialog, select the image you downloaded and hit <code>Continue</code>. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click <code>Import</code>.</p></li> + <li> + <p>After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit <code>Start</code>. Once started up, you will see DrRacket and Firefox on the VM&rsquo;s desktop.</p></li></ol> + + Type-Directed Compilation, Parts I and II + http://prl.ccs.neu.edu/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/?utm_source=Author-Leif-Andersen&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-04-17-type-directed-compilation-parts-i-and-ii + Mon, 17 Apr 2017 12:00:17 UT + Leif Andersen, William J. Bowman + <!-- more--> + +<h3 id="part-i-type-directed-compilation-by-leif-andersen">Part I: <em>Type-Directed Compilation</em>, by Leif Andersen.</h3> + +<p>In this talk we discuss the history of type directed compilation. We start with Xavier Leroy&rsquo;s seminal paper: <a href="http://gallium.inria.fr/~xleroy/publi/unboxed-polymorphism.pdf"><em>Unboxed Objects and Polymorphic Typing</em></a>, continue to <a href="https://www.cs.cmu.edu/~rwh/papers/til/pldi96.pdf">TIL</a> (Typed Intermediate Language), and finish up with <a href="https://dash.harvard.edu/handle/1/2797451">TAL</a> (Typed Assembly Language). We talk about what it means for a compiler to be typed preserving, and give examples of optimizations that are enabled by types.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-24.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;24.md</a></li></ul> + +<h3 id="part-ii-dependent-type-directed-compilation-by-william-j-bowman">Part II: <em>Dependent Type-Directed Compilation</em>, by William J. Bowman</h3> + +<p>A certifying compiler is not verified, but it produces a proof of correctness for each binary. This proof can be independently checked to show that the binary was compiled correctly, removing the compiler from the trusted code base. Certifying compilation has its roots in preserving type-preserving compilation, and in particular in preserving dependent types. We start the history of dependent-type-preserving compilation with a compiler from C to Assembly. We&rsquo;ll see a result showing that preserving dependent types isn&rsquo;t possible, and then we&rsquo;ll do it anyway.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-28.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;28.md</a></li></ul> + +<p>Notes (to appear here, eventually):</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation">https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Li-Yao-Xia.atom.xml b/blog/feeds/Author-Li-Yao-Xia.atom.xml new file mode 100644 index 00000000..549ae356 --- /dev/null +++ b/blog/feeds/Author-Li-Yao-Xia.atom.xml @@ -0,0 +1,289 @@ + + + PRL Blog: Posts tagged 'Author: Li-Yao Xia' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Li-Yao-Xia-html + 2017-06-05T14:27:44Z + + Syntactic parametricity strikes again + + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-syntactic-parametricity-strikes-again + 2017-06-05T14:27:44Z + 2017-06-05T14:27:44Z + Gabriel Scherer + Li-Yao Xia + + Gabriel Scherer, Li-Yao Xia + +<p>In this blog post, reporting on a collaboration with <a href="https://poisson.chat/">Li-Yao Xia</a>, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely &ldquo;syntactic&rdquo; way, by looking at the structure of canonical derivations. More precisely, I prove that \( +\newcommand{\List}[1]{\mathsf{List}~#1} +\newcommand{\Fin}[1]{\mathsf{Fin}~#1} +\newcommand{\Nat}[1]{\mathbb{N}} +\newcommand{\rule}[2]{\frac{\displaystyle \array{#1}}{\displaystyle #2}} +\newcommand{\judge}[2]{{#1} \vdash {#2}} +\newcommand{\emptyrule}[1]{\begin{array}{c}\\[-1em] #1 \end{array}} + ∀α. \List α → \List \alpha +\) is isomorphic to \( + Π(n:\Nat{}). \List{(\Fin{n})} +\) where \(\Fin{n}\) is the type of integers smaller than \(n\), corresponding to the set \(\{0, 1, \dots, n-1\}\).</p> +<!-- more--> + +<p>Context: Last week I had the pleasure of visiting UPenn, where I had many interesting discussions with various people. It was also an occasion to temporarily resume a discussion/collaboration I have with Li-Yao Xia, who is currently an intern there, and Jean-Philippe Bernardy, about testing polymorphic programs and its relation to canonical representations for System F.</p> + +<p>During one of our blackboard discussion, Li-Yao and I did a manual proof of a cool result: we proved a parametricity theorem for \(∀α. \List α → \List α\) using syntactic methods, namely proof search among canonical proofs. (This is an idea that I have been playing with since the last year of my <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/">PhD thesis</a>, where I unsuccessfully tried to extend my work on canonical forms for the simply-typed lambda-calculus to polymorphism. It is here worked out on an specific example, but my end goal is to turn the manual reasoning into an algorithm.)</p> + +<p>You may wonder, first, why the isomorphism holds. The idea is that a polymorphic function of type \(\List α → \List α\) cannot inspect the elements of the input list; it can only use them in the resulting list, possibly duplicating, reordering or dropping some elements. On any input list of size \(n\), the behavior of the function can be described by a list of indices in \([0; n-1]\). For example, if the input \([x, y, z]\) (for some values of \(x, y, z\)) gives the output \([y, y, x]\), then this relation will hold on <em>any</em> value of \(x, y, z\), as the function cannot inspect their value or even test them for equality. The behavior of this function on lists of size 3 can be fully described by the list of indices \([1, 1, 0]\). Its whole behavior is then uniquely determined by one such list for each possible size:</p> + +<p>\[ + ∀α. \List α → \List α \quad≃\quad Π(n:\Nat{}). \List{(\Fin n)} +\]</p> + +<p>The idea behind the &ldquo;syntactic&rdquo; (proof-theoretic?) proof method is the following: the set of closed values at a type \(A\) is isomorphic to the <em>search space</em> for canonical/normal derivations of \(\judge{}{A}\). We have tools (in particular the notion of <em>invertible</em> inference rules) to reason on those – in this post I will only present the reasoning informally, but it can easily be made formally precise.</p> + +<p>We start by looking at the shape of the search space for</p> + +<p>\[ + \judge{}{∀α. \List α → \List α} +\] or, said otherwise, of the judgment \[ + \judge{}{\List α → \List α} +\]</p> + +<p>with a fresh/abstract type variable \(α\). (I will not be keeping opened type variables in context to avoid confusing them with hypotheses.)</p> + +<p>Any derivation of a function type, without loss of generality (w.l.o.g), is equivalent to a derivation starting with a function introduction. This is the η-expansion rule for functions: any proof term \(e\) is equivalent to \(λx.~(e~x)\), a proof that starts with a \(λ\). So any proof can be taken to start as follows: \[ +\rule{ +\judge{\List \alpha}{\List \alpha} +}{ +\judge{}{\List \alpha \to \List \alpha} +} +\] we can, w.l.o.g, unfold the recursive type in the context (\(\List α = 1 + (α × \List α)\)): \[ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}} +\]</p> + +<p>A derivation with a sum type as hypothesis can, w.l.o.g, be assumed to start by splitting on this pair (this is the η-expansion rule for sums): \[ +\rule{ +\judge{1}{\List α} +\quad +\judge{α × \List α}{\List α} +}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]</p> + +<p>In the right subgoal, we can always, w.l.o.g, split a hypothesis of product type: \[ +\rule{ +\emptyrule{\judge{1}{\List α}} +\quad +\rule{ +\judge{α, \List α}{\List α} +}{ +\judge{α × \List α}{\List α} +}}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]</p> + +<p>Now, an interesting pattern emerges. In the process of trying to prove \(\judge{\List α}{\List α}\), we have to prove the (right) subgoal \(\judge{α,\List α}{α}\). We can generalize this derivation by assuming that we start with some number \(n\) of variables of type \(α\) in the context (we write \(α^n\) for this): \[ +\rule{ +\rule{ +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n, 1}{\List \alpha} +} +\quad +\rule{ +\judge{\alpha^{n+1}, \List \alpha}{\List \alpha} +}{ +\judge{\alpha^n, \alpha \times \List \alpha}{\List \alpha} +}}{ +\rule{ +\judge{\alpha^n, 1 + (\alpha \times \List \alpha)}{\List \alpha} +}{ +\judge{\alpha^n, \List \alpha}{\List \alpha} +}} +\]</p> + +<p>\[ +\newcommand{\llbracket}{〚} +\newcommand{\rrbracket}{〛} +\newcommand{\sem}[1]{\llbracket{} #1 \rrbracket{}} +\]</p> + +<p>Let us write \(\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}}\) for the search space corresponding to all possible derivations of the judgment \(\judge{\alpha^n, \List \alpha}{\List \alpha}\). All the proof steps above have been done &ldquo;without loss of generality&rdquo; (in terms of focusing, we only used invertible rules), so they appear in any such derivation. Similarly, let us write \(\sem{\judge{\alpha^n}{\List \alpha}}\) for the space of all possible derivations of \(\judge{\alpha^n}{\List \alpha}\), then above we have proven that \[ +\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} +\quad=\quad +\sem{\judge{\alpha^n}{\List \alpha}} +\times +\sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} +\]</p> + +<p>This equality can be unfolded at will \[ +\begin{align} +&amp; \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+3}, \List \alpha}{\List \alpha}} \\ += &amp; \dots \\ +\end{align} +\]</p> + +<p>or written as an infinite product \[ + \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} + \quad=\quad + \prod_{k \in \Nat{}}{\sem{\judge{\alpha^{n+k}}{\List \alpha}}} +\] and, in particular, \[ +\begin{align} +&amp; \sem{\judge{}{\List \alpha \to \List \alpha}} \\ += &amp; \sem{\judge{\alpha^0, \List \alpha}{\List \alpha}} \\ += &amp; \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ +\end{align} +\]</p> + +<p>Now let&rsquo;s look at the structure of the derivations of \(\judge{\alpha^n}{\List \alpha}\). A proof of this judgment cannot start with a &ldquo;left rule&rdquo;, inspecting the value of one of the \(n\) variables of type \(α\), given that the structure of \(α\) is unknown/abstract. It must start by choosing to either build the empty list or a cons cell. We write this as follows (after unfolding the type):</p> + +<p>\[ +\rule{ +\rule{ +\judge{\alpha^n}{1} +\quad\oplus\quad +\judge{\alpha^n}{\alpha \times \List \alpha} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]</p> + +<p>The \(\oplus\) notation between two judgments is non-standard; it means that they are not two requirements of the same proof, but two alternatives for possible proofs. All valid proofs fit that structure, and they either have a \(\judge{\alpha^n}{1}\) premise or a \(\judge{\alpha^n}{\alpha \times \List \alpha}\) premise. With this syntax, we are describing a set of possible derivations, rather than a single (partial) derivation.</p> + +<p>Proofs of \(\judge{\Gamma}{1}\) are trivial, and a proof of a product is always, w.l.o.g, a product of proofs (in intuitionistic logic / the λ-calculus they reuse the same context), so we can decompose further: \[ +\rule{ +\rule{ +\rule{ +}{ +\judge{\alpha^n}{1} +} +\quad\oplus\quad +\rule +{ +\judge{\alpha^n}{\alpha} +\quad +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n}{\alpha \times \List \alpha} +} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]</p> + +<p>There is exactly one possible proof of \(\judge{\alpha^n}{1}\), so its search space is \(1\), the unit set (with a single element). There are exactly \(n\) possible proofs of \(\judge{\alpha^n}{\alpha}\), so the search space is just \(n\), seen as a set, or, in type-theoretic notation, \(\Fin{n}\). We thus have the recursive equation: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +1 + (\Fin n \times \sem{\judge{\alpha^n}{\List \alpha}}) +\]</p> + +<p>This type is either \(1\), or a \(\Fin{n}\) and itself, recursively. This is exactly a list: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +\List{(\Fin{n})} +\]</p> + +<p>so, plugging everything together: \[ +\begin{align} +&amp; \sem{\forall \alpha. \List \alpha \to \List \alpha} \\ += &amp; \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ += &amp; \prod_{n \in \Nat{}}{\List{(\Fin{n})}} \\ +\end{align} +\]</p> + +<h3 id="post-scriptum">Post Scriptum</h3> + +<p>Some of reasoning steps above can be formulated in a way that is less clear but more familiar, as a sequence of type isomorphisms. For example, the first part on \(\sem{\judge{\alpha^n, \List +\alpha}{\List \alpha}}\) can written as:</p> + +<p>\[ +\begin{align} +&amp; +∀α. αⁿ × \List α → \List α +\\ &amp; += \text{(unfold List)} +\\ &amp; + ∀α. αⁿ × (1 + α × \List α) → \List α +\\ &amp; + = \text{(distribute × over +)} +\\ &amp; + ∀α. ((αⁿ × 1) + (αⁿ⁺¹ × \List α)) → \List α +\\ &amp; + = \text{(A × 1 ≃ A)} +\\ &amp; + ∀α. (αⁿ + (αⁿ⁺¹ × \List α)) → \List α +\\ &amp; + = \text{(A+B) → C ≃ (A→C)×(B→C)} +\\ &amp; + ∀α. (αⁿ → \List α) × (αⁿ⁺¹ × \List α → \List α) +\\ &amp; + = \text{(distribute ∀α below product)} +\\ &amp; + (∀α. αⁿ → \List α) × (∀α. αⁿ⁺¹ × \List α → \List α) +\\ +\end{align} +\]</p> + +<p>Reading this equational sequence, it may look like we had to make the right choice at each step; but the proof-search perspective reveals that there were in fact no choices, as each time we apply invertible rules (&ldquo;w.l.o.g. rules&rdquo;).</p> + +<p>Furthermore, some parts cannot be derived in this style; in the latter part of the proof, the isomorphism between \(∀\alpha. \alpha^n → \alpha\) and \(\Fin{n}\), which is immediate from a proof search perspective, cannot be justified in this way. (In particular, \(A×B → C\) is <em>not</em> isomorphic to \((A→C)+(B→C)\).)</p> + +<h3 id="going-further">Going further</h3> + +<ul> + <li> + <p>It is an unfortunately-little-known obvious fact that many things we associate to &ldquo;free theorems&rdquo; can be recovered by proof search. For example, it is much simpler to prove that the only inhabitant of \(\forall \alpha. \alpha \to \alpha\) is the identity using proof search than parametricity. I briefly discussed the idea in the section 1.3 of my 2015 article, <a href="http://gallium.inria.fr/~scherer/research/unique_inhabitants/unique_stlc_sums-long.pdf">Which simple types have a unique inhabitant?</a>.</p></li> + <li> + <p>If you are unfamiliar with proof search (or the LF community) and curious about what I mean by &ldquo;canonical forms&rdquo; and why I think this is an important idea, see my non-technical 2017 article <a href="http://www.ccs.neu.edu/home/gasche/research/canonical-forms/snapl.pdf">Search for Program Structure</a>. The problem of extending the notion of canonical forms to arbitrary polymorphic types is briefly discussed in the section 14.5 of my 2016 <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">phd manuscript</a>.</p></li> + <li> + <p>If you haven&rsquo;t heard of it yet, you would probably be interested in the 2010 article <a href="http://publications.lib.chalmers.se/records/fulltext/local_99387.pdf">Testing Polymorphic Properties</a> by Jean-Philippe Bernardy, Patrik Jansson and Koen Claessen. Li-Yao has a 2016 implementation called <a href="https://github.com/Lysxia/metamorph">Metamorph</a> that got us talking together. The problems of understanding canonical forms and testing are quite related, but yet not exactly the same&hellip;</p></li></ul> + +<h3 id="you-might-also-like">You might also like</h3> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="https://williamjbowman.com/blog/2017/01/03/toward-type-preserving-compilation-of-coq-at-popl17-src/">Toward Type-Preserving Compilation of Coq, at POPL17 SRC</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Li-Yao-Xia.rss.xml b/blog/feeds/Author-Li-Yao-Xia.rss.xml new file mode 100644 index 00000000..2148f7ff --- /dev/null +++ b/blog/feeds/Author-Li-Yao-Xia.rss.xml @@ -0,0 +1,287 @@ + + + + PRL Blog: Posts tagged 'Author: Li-Yao Xia' + PRL Blog: Posts tagged 'Author: Li-Yao Xia' + http://prl.ccs.neu.edu/blog/tags/Author-Li-Yao-Xia.html + Mon, 05 Jun 2017 14:27:44 UT + Mon, 05 Jun 2017 14:27:44 UT + 1800 + + Syntactic parametricity strikes again + http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/?utm_source=Author-Li-Yao-Xia&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-syntactic-parametricity-strikes-again + Mon, 05 Jun 2017 14:27:44 UT + Gabriel Scherer, Li-Yao Xia + +<p>In this blog post, reporting on a collaboration with <a href="https://poisson.chat/">Li-Yao Xia</a>, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely &ldquo;syntactic&rdquo; way, by looking at the structure of canonical derivations. More precisely, I prove that \( +\newcommand{\List}[1]{\mathsf{List}~#1} +\newcommand{\Fin}[1]{\mathsf{Fin}~#1} +\newcommand{\Nat}[1]{\mathbb{N}} +\newcommand{\rule}[2]{\frac{\displaystyle \array{#1}}{\displaystyle #2}} +\newcommand{\judge}[2]{{#1} \vdash {#2}} +\newcommand{\emptyrule}[1]{\begin{array}{c}\\[-1em] #1 \end{array}} + ∀α. \List α → \List \alpha +\) is isomorphic to \( + Π(n:\Nat{}). \List{(\Fin{n})} +\) where \(\Fin{n}\) is the type of integers smaller than \(n\), corresponding to the set \(\{0, 1, \dots, n-1\}\).</p> +<!-- more--> + +<p>Context: Last week I had the pleasure of visiting UPenn, where I had many interesting discussions with various people. It was also an occasion to temporarily resume a discussion/collaboration I have with Li-Yao Xia, who is currently an intern there, and Jean-Philippe Bernardy, about testing polymorphic programs and its relation to canonical representations for System F.</p> + +<p>During one of our blackboard discussion, Li-Yao and I did a manual proof of a cool result: we proved a parametricity theorem for \(∀α. \List α → \List α\) using syntactic methods, namely proof search among canonical proofs. (This is an idea that I have been playing with since the last year of my <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/">PhD thesis</a>, where I unsuccessfully tried to extend my work on canonical forms for the simply-typed lambda-calculus to polymorphism. It is here worked out on an specific example, but my end goal is to turn the manual reasoning into an algorithm.)</p> + +<p>You may wonder, first, why the isomorphism holds. The idea is that a polymorphic function of type \(\List α → \List α\) cannot inspect the elements of the input list; it can only use them in the resulting list, possibly duplicating, reordering or dropping some elements. On any input list of size \(n\), the behavior of the function can be described by a list of indices in \([0; n-1]\). For example, if the input \([x, y, z]\) (for some values of \(x, y, z\)) gives the output \([y, y, x]\), then this relation will hold on <em>any</em> value of \(x, y, z\), as the function cannot inspect their value or even test them for equality. The behavior of this function on lists of size 3 can be fully described by the list of indices \([1, 1, 0]\). Its whole behavior is then uniquely determined by one such list for each possible size:</p> + +<p>\[ + ∀α. \List α → \List α \quad≃\quad Π(n:\Nat{}). \List{(\Fin n)} +\]</p> + +<p>The idea behind the &ldquo;syntactic&rdquo; (proof-theoretic?) proof method is the following: the set of closed values at a type \(A\) is isomorphic to the <em>search space</em> for canonical/normal derivations of \(\judge{}{A}\). We have tools (in particular the notion of <em>invertible</em> inference rules) to reason on those – in this post I will only present the reasoning informally, but it can easily be made formally precise.</p> + +<p>We start by looking at the shape of the search space for</p> + +<p>\[ + \judge{}{∀α. \List α → \List α} +\] or, said otherwise, of the judgment \[ + \judge{}{\List α → \List α} +\]</p> + +<p>with a fresh/abstract type variable \(α\). (I will not be keeping opened type variables in context to avoid confusing them with hypotheses.)</p> + +<p>Any derivation of a function type, without loss of generality (w.l.o.g), is equivalent to a derivation starting with a function introduction. This is the η-expansion rule for functions: any proof term \(e\) is equivalent to \(λx.~(e~x)\), a proof that starts with a \(λ\). So any proof can be taken to start as follows: \[ +\rule{ +\judge{\List \alpha}{\List \alpha} +}{ +\judge{}{\List \alpha \to \List \alpha} +} +\] we can, w.l.o.g, unfold the recursive type in the context (\(\List α = 1 + (α × \List α)\)): \[ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}} +\]</p> + +<p>A derivation with a sum type as hypothesis can, w.l.o.g, be assumed to start by splitting on this pair (this is the η-expansion rule for sums): \[ +\rule{ +\judge{1}{\List α} +\quad +\judge{α × \List α}{\List α} +}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]</p> + +<p>In the right subgoal, we can always, w.l.o.g, split a hypothesis of product type: \[ +\rule{ +\emptyrule{\judge{1}{\List α}} +\quad +\rule{ +\judge{α, \List α}{\List α} +}{ +\judge{α × \List α}{\List α} +}}{ +\rule{ +\judge{1 + (α × \List α)}{\List α} +}{ +\rule{ +\judge{\List α}{\List α} +}{ +\judge{}{\List α → \List α} +}}} +\]</p> + +<p>Now, an interesting pattern emerges. In the process of trying to prove \(\judge{\List α}{\List α}\), we have to prove the (right) subgoal \(\judge{α,\List α}{α}\). We can generalize this derivation by assuming that we start with some number \(n\) of variables of type \(α\) in the context (we write \(α^n\) for this): \[ +\rule{ +\rule{ +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n, 1}{\List \alpha} +} +\quad +\rule{ +\judge{\alpha^{n+1}, \List \alpha}{\List \alpha} +}{ +\judge{\alpha^n, \alpha \times \List \alpha}{\List \alpha} +}}{ +\rule{ +\judge{\alpha^n, 1 + (\alpha \times \List \alpha)}{\List \alpha} +}{ +\judge{\alpha^n, \List \alpha}{\List \alpha} +}} +\]</p> + +<p>\[ +\newcommand{\llbracket}{〚} +\newcommand{\rrbracket}{〛} +\newcommand{\sem}[1]{\llbracket{} #1 \rrbracket{}} +\]</p> + +<p>Let us write \(\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}}\) for the search space corresponding to all possible derivations of the judgment \(\judge{\alpha^n, \List \alpha}{\List \alpha}\). All the proof steps above have been done &ldquo;without loss of generality&rdquo; (in terms of focusing, we only used invertible rules), so they appear in any such derivation. Similarly, let us write \(\sem{\judge{\alpha^n}{\List \alpha}}\) for the space of all possible derivations of \(\judge{\alpha^n}{\List \alpha}\), then above we have proven that \[ +\sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} +\quad=\quad +\sem{\judge{\alpha^n}{\List \alpha}} +\times +\sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} +\]</p> + +<p>This equality can be unfolded at will \[ +\begin{align} +&amp; \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}, \List \alpha}{\List \alpha}} \\ += &amp; \sem{\judge{\alpha^n}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+1}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+2}}{\List \alpha}} + \times + \sem{\judge{\alpha^{n+3}, \List \alpha}{\List \alpha}} \\ += &amp; \dots \\ +\end{align} +\]</p> + +<p>or written as an infinite product \[ + \sem{\judge{\alpha^n, \List \alpha}{\List \alpha}} + \quad=\quad + \prod_{k \in \Nat{}}{\sem{\judge{\alpha^{n+k}}{\List \alpha}}} +\] and, in particular, \[ +\begin{align} +&amp; \sem{\judge{}{\List \alpha \to \List \alpha}} \\ += &amp; \sem{\judge{\alpha^0, \List \alpha}{\List \alpha}} \\ += &amp; \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ +\end{align} +\]</p> + +<p>Now let&rsquo;s look at the structure of the derivations of \(\judge{\alpha^n}{\List \alpha}\). A proof of this judgment cannot start with a &ldquo;left rule&rdquo;, inspecting the value of one of the \(n\) variables of type \(α\), given that the structure of \(α\) is unknown/abstract. It must start by choosing to either build the empty list or a cons cell. We write this as follows (after unfolding the type):</p> + +<p>\[ +\rule{ +\rule{ +\judge{\alpha^n}{1} +\quad\oplus\quad +\judge{\alpha^n}{\alpha \times \List \alpha} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]</p> + +<p>The \(\oplus\) notation between two judgments is non-standard; it means that they are not two requirements of the same proof, but two alternatives for possible proofs. All valid proofs fit that structure, and they either have a \(\judge{\alpha^n}{1}\) premise or a \(\judge{\alpha^n}{\alpha \times \List \alpha}\) premise. With this syntax, we are describing a set of possible derivations, rather than a single (partial) derivation.</p> + +<p>Proofs of \(\judge{\Gamma}{1}\) are trivial, and a proof of a product is always, w.l.o.g, a product of proofs (in intuitionistic logic / the λ-calculus they reuse the same context), so we can decompose further: \[ +\rule{ +\rule{ +\rule{ +}{ +\judge{\alpha^n}{1} +} +\quad\oplus\quad +\rule +{ +\judge{\alpha^n}{\alpha} +\quad +\judge{\alpha^n}{\List \alpha} +}{ +\judge{\alpha^n}{\alpha \times \List \alpha} +} +}{ +\judge{\alpha^n}{1 + (\alpha \times \List \alpha)} +}}{ +\judge{\alpha^n}{\List \alpha} +} +\]</p> + +<p>There is exactly one possible proof of \(\judge{\alpha^n}{1}\), so its search space is \(1\), the unit set (with a single element). There are exactly \(n\) possible proofs of \(\judge{\alpha^n}{\alpha}\), so the search space is just \(n\), seen as a set, or, in type-theoretic notation, \(\Fin{n}\). We thus have the recursive equation: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +1 + (\Fin n \times \sem{\judge{\alpha^n}{\List \alpha}}) +\]</p> + +<p>This type is either \(1\), or a \(\Fin{n}\) and itself, recursively. This is exactly a list: \[ +\sem{\judge{\alpha^n}{\List \alpha}} +\quad=\quad +\List{(\Fin{n})} +\]</p> + +<p>so, plugging everything together: \[ +\begin{align} +&amp; \sem{\forall \alpha. \List \alpha \to \List \alpha} \\ += &amp; \prod_{n \in \Nat{}}{\sem{\judge{\alpha^n}{\List \alpha}}} \\ += &amp; \prod_{n \in \Nat{}}{\List{(\Fin{n})}} \\ +\end{align} +\]</p> + +<h3 id="post-scriptum">Post Scriptum</h3> + +<p>Some of reasoning steps above can be formulated in a way that is less clear but more familiar, as a sequence of type isomorphisms. For example, the first part on \(\sem{\judge{\alpha^n, \List +\alpha}{\List \alpha}}\) can written as:</p> + +<p>\[ +\begin{align} +&amp; +∀α. αⁿ × \List α → \List α +\\ &amp; += \text{(unfold List)} +\\ &amp; + ∀α. αⁿ × (1 + α × \List α) → \List α +\\ &amp; + = \text{(distribute × over +)} +\\ &amp; + ∀α. ((αⁿ × 1) + (αⁿ⁺¹ × \List α)) → \List α +\\ &amp; + = \text{(A × 1 ≃ A)} +\\ &amp; + ∀α. (αⁿ + (αⁿ⁺¹ × \List α)) → \List α +\\ &amp; + = \text{(A+B) → C ≃ (A→C)×(B→C)} +\\ &amp; + ∀α. (αⁿ → \List α) × (αⁿ⁺¹ × \List α → \List α) +\\ &amp; + = \text{(distribute ∀α below product)} +\\ &amp; + (∀α. αⁿ → \List α) × (∀α. αⁿ⁺¹ × \List α → \List α) +\\ +\end{align} +\]</p> + +<p>Reading this equational sequence, it may look like we had to make the right choice at each step; but the proof-search perspective reveals that there were in fact no choices, as each time we apply invertible rules (&ldquo;w.l.o.g. rules&rdquo;).</p> + +<p>Furthermore, some parts cannot be derived in this style; in the latter part of the proof, the isomorphism between \(∀\alpha. \alpha^n → \alpha\) and \(\Fin{n}\), which is immediate from a proof search perspective, cannot be justified in this way. (In particular, \(A×B → C\) is <em>not</em> isomorphic to \((A→C)+(B→C)\).)</p> + +<h3 id="going-further">Going further</h3> + +<ul> + <li> + <p>It is an unfortunately-little-known obvious fact that many things we associate to &ldquo;free theorems&rdquo; can be recovered by proof search. For example, it is much simpler to prove that the only inhabitant of \(\forall \alpha. \alpha \to \alpha\) is the identity using proof search than parametricity. I briefly discussed the idea in the section 1.3 of my 2015 article, <a href="http://gallium.inria.fr/~scherer/research/unique_inhabitants/unique_stlc_sums-long.pdf">Which simple types have a unique inhabitant?</a>.</p></li> + <li> + <p>If you are unfamiliar with proof search (or the LF community) and curious about what I mean by &ldquo;canonical forms&rdquo; and why I think this is an important idea, see my non-technical 2017 article <a href="http://www.ccs.neu.edu/home/gasche/research/canonical-forms/snapl.pdf">Search for Program Structure</a>. The problem of extending the notion of canonical forms to arbitrary polymorphic types is briefly discussed in the section 14.5 of my 2016 <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">phd manuscript</a>.</p></li> + <li> + <p>If you haven&rsquo;t heard of it yet, you would probably be interested in the 2010 article <a href="http://publications.lib.chalmers.se/records/fulltext/local_99387.pdf">Testing Polymorphic Properties</a> by Jean-Philippe Bernardy, Patrik Jansson and Koen Claessen. Li-Yao has a 2016 implementation called <a href="https://github.com/Lysxia/metamorph">Metamorph</a> that got us talking together. The problems of understanding canonical forms and testing are quite related, but yet not exactly the same&hellip;</p></li></ul> + +<h3 id="you-might-also-like">You might also like</h3> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="https://williamjbowman.com/blog/2017/01/03/toward-type-preserving-compilation-of-coq-at-popl17-src/">Toward Type-Preserving Compilation of Coq, at POPL17 SRC</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Matt-Kolosick.atom.xml b/blog/feeds/Author-Matt-Kolosick.atom.xml new file mode 100644 index 00000000..e6f2ac65 --- /dev/null +++ b/blog/feeds/Author-Matt-Kolosick.atom.xml @@ -0,0 +1,84 @@ + + + PRL Blog: Posts tagged 'Author: Matt Kolosick' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Matt-Kolosick-html + 2017-06-16T11:38:25Z + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Matt-Kolosick.rss.xml b/blog/feeds/Author-Matt-Kolosick.rss.xml new file mode 100644 index 00000000..82c98d38 --- /dev/null +++ b/blog/feeds/Author-Matt-Kolosick.rss.xml @@ -0,0 +1,76 @@ + + + + PRL Blog: Posts tagged 'Author: Matt Kolosick' + PRL Blog: Posts tagged 'Author: Matt Kolosick' + http://prl.ccs.neu.edu/blog/tags/Author-Matt-Kolosick.html + Fri, 16 Jun 2017 11:38:25 UT + Fri, 16 Jun 2017 11:38:25 UT + 1800 + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=Author-Matt-Kolosick&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Max-New.atom.xml b/blog/feeds/Author-Max-New.atom.xml new file mode 100644 index 00000000..a56a676e --- /dev/null +++ b/blog/feeds/Author-Max-New.atom.xml @@ -0,0 +1,535 @@ + + + PRL Blog: Posts tagged 'Author: Max New' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Max-New-html + 2017-09-27T15:44:57Z + + Final Algebra Semantics is Observational Equivalence + + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + 2017-09-27T15:44:57Z + 2017-09-27T15:44:57Z + + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> + + Closure Conversion as CoYoneda + + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + 2017-08-28T10:30:00Z + 2017-08-28T10:30:00Z + + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> + + Categorical Semantics for Dynamically Typed Programming Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + 2017-05-01T12:25:17Z + 2017-05-01T12:25:17Z + + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> + + Understanding Constructive Galois Connections + + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + 2016-11-16T00:00:00Z + 2016-11-16T00:00:00Z + + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Max-New.rss.xml b/blog/feeds/Author-Max-New.rss.xml new file mode 100644 index 00000000..467f52cd --- /dev/null +++ b/blog/feeds/Author-Max-New.rss.xml @@ -0,0 +1,529 @@ + + + + PRL Blog: Posts tagged 'Author: Max New' + PRL Blog: Posts tagged 'Author: Max New' + http://prl.ccs.neu.edu/blog/tags/Author-Max-New.html + Wed, 27 Sep 2017 15:44:57 UT + Wed, 27 Sep 2017 15:44:57 UT + 1800 + + Final Algebra Semantics is Observational Equivalence + http://prl.ccs.neu.edu/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/?utm_source=Author-Max-New&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + Wed, 27 Sep 2017 15:44:57 UT + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> + + Closure Conversion as CoYoneda + http://prl.ccs.neu.edu/blog/2017/08/28/closure-conversion-as-coyoneda/?utm_source=Author-Max-New&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + Mon, 28 Aug 2017 10:30:00 UT + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> + + Categorical Semantics for Dynamically Typed Programming Languages + http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/?utm_source=Author-Max-New&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + Mon, 01 May 2017 12:25:17 UT + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> + + Understanding Constructive Galois Connections + http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/?utm_source=Author-Max-New&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + Wed, 16 Nov 2016 00:00:00 UT + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Milo-Davis.atom.xml b/blog/feeds/Author-Milo-Davis.atom.xml new file mode 100644 index 00000000..14778006 --- /dev/null +++ b/blog/feeds/Author-Milo-Davis.atom.xml @@ -0,0 +1,426 @@ + + + PRL Blog: Posts tagged 'Author: Milo Davis' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Milo-Davis-html + 2017-06-16T11:38:25Z + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Fall 2016 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + 2017-01-02T16:39:37Z + 2017-01-02T16:39:37Z + Ben Chung + Milo Davis + Ming-Ho Yee + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> + + Beta Reduction (Part 1) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + 2016-11-02T21:10:18Z + 2016-11-02T21:10:18Z + + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Milo-Davis.rss.xml b/blog/feeds/Author-Milo-Davis.rss.xml new file mode 100644 index 00000000..93fe1a36 --- /dev/null +++ b/blog/feeds/Author-Milo-Davis.rss.xml @@ -0,0 +1,410 @@ + + + + PRL Blog: Posts tagged 'Author: Milo Davis' + PRL Blog: Posts tagged 'Author: Milo Davis' + http://prl.ccs.neu.edu/blog/tags/Author-Milo-Davis.html + Fri, 16 Jun 2017 11:38:25 UT + Fri, 16 Jun 2017 11:38:25 UT + 1800 + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=Author-Milo-Davis&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Fall 2016 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/?utm_source=Author-Milo-Davis&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + Mon, 02 Jan 2017 16:39:37 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> + + Beta Reduction (Part 1) + http://prl.ccs.neu.edu/blog/2016/11/02/beta-reduction-part-1/?utm_source=Author-Milo-Davis&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + Wed, 02 Nov 2016 21:10:18 UT + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Ming-Ho-Yee.atom.xml b/blog/feeds/Author-Ming-Ho-Yee.atom.xml new file mode 100644 index 00000000..255ee45c --- /dev/null +++ b/blog/feeds/Author-Ming-Ho-Yee.atom.xml @@ -0,0 +1,1242 @@ + + + PRL Blog: Posts tagged 'Author: Ming-Ho Yee' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Ming-Ho-Yee-html + 2019-09-10T11:00:00Z + + Four Kinds of Scoping in R + + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-four-kinds-of-scoping-in-r + 2019-09-10T11:00:00Z + 2019-09-10T11:00:00Z + + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">first</a> and <a href="/blog/2019/09/10/scoping-in-r/">second</a> parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.</p> + +<p>In this third and final part of my blog series, I&rsquo;d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.</p> + +<p>This is a &ldquo;bonus&rdquo; blog post, because I&rsquo;m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.</p> +<!-- more--> + +<h2 id="lexical-scope-and-statistical-computation">Lexical scope and statistical computation</h2> + +<p>In <em>Lexical Scope and Statistical Computation</em>,<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">1</a></sup> Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.</p> + +<p>For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:</p> + +<ul> + <li><em>trivial</em>: no free variables allowed</li> + <li><em>static</em>: a free variable takes its value from a set of global variables</li> + <li><em>lexical</em>: a free variable takes the value of the binding that was in effect when the function was defined</li> + <li><em>dynamic</em>: a free variable takes the value of the most recent assignment to that variable</li></ul> + +<p>Note that under this set of definitions, <em>static scoping</em> is a separate scoping rule and not another name for <em>lexical scoping</em>.</p> + +<p>It is possible to simulate each of strategies in R. For fun, we can even construct &ldquo;factories&rdquo; that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)</p> + +<h3 id="template">Template</h3> + +<p>Our examples will follow the template given below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">factory</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="o">&lt;???&gt;</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">factory</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># error, 0, 1, or 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We want to define a <code>factory</code> that takes a function literal and returns a closure that implements the desired scoping rule.</p> + +<p>Our example consists of three definitions of <code>x</code>. On line 5, we assign <code>0</code> to <code>x</code> at the top level. On line 7, we assign <code>1</code> to <code>x</code> inside function <code>h</code>, where we also create the closure. On line 12, we assign <code>2</code> to <code>x</code> inside the function <code>f</code> and right before we call <code>g</code>, which is the closure.</p> + +<p>Finally, we call <code>f</code> and observe the result:</p> + +<ul> + <li>Under trivial scoping, no free variables are allowed, so <code>f()</code> should result in an error.</li> + <li>Under static scoping, free variables may only refer to global variables, so <code>f()</code> should return <code>0</code>.</li> + <li>Under lexical scoping, free variables refer to the variables in scope when the function was defined, so <code>f()</code> should return <code>1</code>.</li> + <li>Under dynamic scoping, free variables take the value from the most recent assignment, so <code>f()</code> should return <code>2</code>.</li></ul> + +<p>We will implement the body of <code>factory</code> in only 3&ndash;5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to <code>factory</code> on line 10.</p> + +<h3 id="trivial-scoping">Trivial scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeTrivial</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">baseenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeTrivial</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># Error in f(0) : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>substitute</code> returns the unevaluated parse tree for <code>fun</code>. In other words, it obtains the literal argument that was passed for <code>fun</code>. This works because of call-by-need semantics in R: function arguments are packaged up into <em>promises</em>. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">2</a></sup> discusses laziness in R in more detail.</p> + +<p>In this example, on line 8, we call <code>factory</code> with <code>function(a) x+a</code> as the argument for the formal parameter <code>fun</code>. Then, we evaluate that parse tree with <code>eval</code>.</p> + +<p>At this point, <code>res</code> is the closure with expression <code>function(a) x+a</code> and a reference to the environment of <code>makeTrivial</code>. On line 3, we change that reference to <code>baseenv()</code>, which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.</p> + +<p>Therefore, variable lookup in the function literal will only search the base environment, so <code>f()</code> results in an error.</p> + +<h3 id="static-scoping">Static scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeStatic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">globalenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeStatic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, on line 3, we update the environment of <code>res</code> to refer to <code>globalenv()</code>, which is the top-level environment where globals are defined.</p> + +<p>Therefore, variable lookup searches the top-level environment, so <code>f()</code> returns <code>0</code>.</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeLexical</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeLexical</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of <code>res</code> to <code>parent.frame()</code>, which is the environment of the function (<code>h</code>) that called the current function (<code>makeLexical</code>). This allows us to simulate lexical scoping, as if the function literal was evaluated inside <code>h</code>, rather than <code>makeLexical</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>h</code>, so <code>f()</code> returns <code>1</code>.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeDynamic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">function</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="nf">res</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> + <span class="p">}</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeDynamic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, we need another level of indirection. <code>makeDynamic</code> returns an anonymous function literal. The anonymous function takes <code>...</code>, which represents an arbitrary list of arguments, and then on line 5 we call <code>res</code> with those exact arguments. Note that we set the environment of <code>res</code> to be the environment of the <em>caller</em> of the anonymous function. Because of the multiple levels of indirection, the caller is <code>f</code>, on line 17.</p> + +<p>On line 12, <code>makeDynamic</code> returns a closure for the anonymous function. <code>h</code> returns that closure when it is called, and assigns it to <code>g</code>. When <code>g</code> is called on line 17, the function literal <code>function(a) x+a</code> is finally evaluated, and its environment is set to the environment of <code>f</code>, the caller of <code>g</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>f</code>, so <code>f()</code> returns <code>2</code>.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the <a href="/blog/2019/09/10/scoping-in-r/">previous post</a>, it&rsquo;s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.</p> + +<p>And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.</p> + +<p><em>I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>A. Goel and J. Vitek. &ldquo;On the Design, Implementation and Use of Laziness in R,&rdquo; in <em>Proceedings of the ACM in Programming Languages (PACMPL)</em>, vol. 3, no. OOPSLA, 2019. To appear. [<a href="http://janvitek.org/pubs/oopsla19a.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">↩</a></p></li></ol></div> + + Scoping in R + + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-scoping-in-r + 2019-09-10T10:00:00Z + 2019-09-10T10:00:00Z + + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">previous post</a> of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: <em>is R lexically or dynamically scoped?</em></p> +<!-- more--> + +<p>Recall the example program from before:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s examine what happens when we run this example. First, we create a mapping for <code>x</code> in the top-level environment. On line 2, we define a function <code>f</code>, which returns the value of some <code>x</code>. On line 3, we define a function <code>g</code>, which creates a new mapping for <code>x</code>, and then calls <code>f</code>. Note that the assignment on line 4 does <em>not</em> update the definition on line 1.</p> + +<p>When <code>f</code> is called, it needs to look up the value of <code>x</code>. In other words, does the reference of <code>x</code> on line 2 refer to the assignment on line 1 or the assignment on line 4? If <code>f</code> returns <code>1</code>, then the behaviour matches lexical scoping. If it returns <code>2</code>, then the behaviour matches dynamic scoping.</p> + +<p>When we run this example, the result is <code>1</code>. This implies that R is lexically scoped.</p> + +<p>But there&rsquo;s more to this story. In the rest of this blog post, I&rsquo;ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.</p> + +<p>The <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">next and final part</a> of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.</p> + +<h2 id="r-is-lexically-scoped-but">R is lexically scoped, but&hellip;</h2> + +<p>In <em>Evaluating the Design of the R Language</em>,<sup><a href="#2019-09-10-scoping-in-r-footnote-1-definition" name="2019-09-10-scoping-in-r-footnote-1-return">1</a></sup> Morandat, Hill, Osvald, and Vitek write:</p> + +<blockquote> + <p>As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.</p></blockquote> + +<p>In other words, R provides many different &ldquo;escape hatches&rdquo;&mdash;ways to bypass lexical scoping. Additionally, even without escape hatches, some of R&rsquo;s functionality can be surprising.</p> + +<h3 id="functions-environments-and-variables-in-r">Functions, environments, and variables in R</h3> + +<p>Before we look at some examples, I think it&rsquo;s useful to briefly discuss some of the core concepts in R that relate to scoping.</p> + +<ul> + <li> + <p><strong>Functions.</strong> R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.</p></li> + <li> + <p><strong>Environments.</strong> An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the &ldquo;enclosing&rdquo; environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.</p></li> + <li> + <p><strong>Variable lookup.</strong> When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.</p></li> + <li> + <p><strong>Variable assignment.</strong> <code>&lt;-</code> is the variable assignment operator in R. The expression <code>x &lt;- 1</code> assigns the value <code>1</code> to the variable <code>x</code> in the current environment. If a mapping for <code>x</code> already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.</p></li></ul> + +<p>From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.<sup><a href="#2019-09-10-scoping-in-r-footnote-2-definition" name="2019-09-10-scoping-in-r-footnote-2-return">2</a></sup></p> + +<p>On the other hand, variable lookup depends on the run-time state of the program&mdash;names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.</p> + +<p>The following examples will make this clear.</p> + +<h3 id="examples">Examples</h3> + +<h4 id="adding-variable-mappings-at-run-time">Adding variable mappings at run time</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it creates a function <code>g</code> that returns <code>x</code>, assigns <code>2</code> to <code>x</code>, and then calls <code>g</code>. When <code>g</code> is called, it looks up <code>x</code>. Since no mapping is found in <code>g</code>&rsquo;s environment, it searches in the enclosing environment (<code>f</code>&rsquo;s), and finds that <code>x</code> has value <code>2</code>. Therefore, <code>g</code> returns <code>2</code>.</p> + +<p>Note that the <code>x</code> on line 3 is resolved only when function <code>g</code> is called, not when it is defined. However, when <code>g</code> is defined, its environment has a reference to <code>f</code>&rsquo;s environment. Therefore, as long as <code>x</code> is defined <em>before</em> <code>g</code> is called, the lookup will always succeed.</p> + +<p>Here&rsquo;s a second example:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>f</code> is a function that branches on its argument, <code>b</code>. If <code>b</code> evaluates to true, then the expression <code>x &lt;- 2</code> is evaluated, and a mapping for <code>x</code> is created in <code>f</code>&rsquo;s environment. Otherwise, no mapping is created.</p> + +<p>When we look up the value of <code>x</code> on line 5, R will first search the function&rsquo;s environment. If <code>b</code> evaluated to true, then R will find a value for <code>x</code>, which is <code>2</code>. Otherwise, R will search in the enclosing environment of <code>f</code>, and find that <code>x</code> is <code>1</code>.</p> + +<p>Both of these examples vaguely resemble dynamic scoping, in that <code>x</code> takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.</p> + +<h4 id="function-lookup">Function lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> <span class="c1"># not an error</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">(</span><span class="m">42</span><span class="p">)</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.</p> + +<p>In this example, we call <code>g</code> with the argument <code>42</code>, which is not a function. Then, in the body of <code>g</code>, we call <code>f(0)</code> on line 3, which requires looking up <code>f</code>. Although there is an <code>f</code> in the environment of <code>g</code>, its value is <code>42</code>, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so <code>f(0)</code> returns <code>0</code>.</p> + +<p>This behaviour exists because <code>c</code> is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.</p> + +<h4 id="super-assignment">Super assignment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="n">x</span> <span class="o">&lt;&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +<span class="n">x</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>&lt;&lt;-</code> is the &ldquo;super assignment&rdquo; operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.</p> + +<p>In the above program, we define <code>x</code> to be <code>0</code> at the top level, and then define the function <code>f</code>. When we call <code>f</code> on line 7, it assigns <code>1</code> to <code>x</code> on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, <code>f</code> returns <code>x</code>, which is looked up from the local environment and has value <code>1</code>. Finally, line 8 looks up <code>x</code> from the top level environment, which has value <code>2</code>.</p> + +<h4 id="evaluating-arbitrary-code">Evaluating arbitrary code</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">eval</span><span class="p">(</span><span class="nf">parse</span><span class="p">(</span><span class="n">text</span> <span class="o">=</span> <span class="n">t</span><span class="p">))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="s">"x &lt;- 0"</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument <code>t</code>, which happens to be the string <code>"x &lt;- 0"</code>. Then, when line 4 executes, the lookup of <code>x</code> returns <code>0</code>.</p> + +<h4 id="simulating-dynamic-scope">Simulating dynamic scope</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.frame</span><span class="p">())</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 3, we perform an explicit variable lookup for <code>x</code>, but we do so in the environment <code>parent.frame()</code>, which refers to the calling function&rsquo;s environment, in this case, <code>g</code>&rsquo;s environment.. Therefore, the lookup returns <code>2</code>.</p> + +<p>Note that R has a similarly named function, <code>parent.env(e)</code> which returns the <em>enclosing environment</em> of the given environment <code>e</code>.</p> + +<h4 id="constructing-an-arbitrary-environment">Constructing an arbitrary environment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">e</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> + <span class="n">e</span><span class="o">$</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">3</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="n">e</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 3</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it constructs a new environment, <code>e</code>, which is initially empty. (By default, its enclosing environment is the current environment, which is <code>f</code>&rsquo;s.) Next, on line 4, it directly adds a mapping to that environment, assigning <code>3</code> to <code>x</code>. Then, on line 5, the lookup is explicitly done in environment <code>e</code>, so <code>f</code> returns <code>3</code>.</p> + +<h4 id="deleting-mappings">Deleting mappings</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">rm</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.env</span><span class="p">(</span><span class="nf">environment</span><span class="p">()))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># Error in f() : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Not only is it possible to dynamically add and modify mappings in R, but it is also possible to <em>delete</em> mappings. This is what line 3 does: it explicitly removes the mapping for <code>x</code> from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when <code>f</code> is called, the lookup of <code>x</code> fails and an error is raised.</p> + +<h4 id="infinite-loop-during-variable-lookup">Infinite loop during variable lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">enva</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="n">envb</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">enva</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">envb</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">envb</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="nf">environment</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="nf">f</span><span class="p">()</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.</p> + +<p>On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: <code>enva</code>&rsquo;s enclosing environment is <code>envb</code>, and <code>envb</code>&rsquo;s enclosing environment is <code>enva</code>.</p> + +<p>On line 5, we define a function with a free variable, <code>x</code>, but on line 6, we set <code>f</code>&rsquo;s environment to be <code>enva</code>. Finally, we call <code>f</code>.</p> + +<p>When the body of <code>f</code> is evaluated, it needs to look up <code>x</code>. Lookup starts in <code>f</code>&rsquo;s environment, which we set to be <code>enva</code>. Since no mapping for <code>x</code> is found, lookup continues in <code>enva</code>&rsquo;s enclosing environment, which is <code>envb</code>. However, <code>envb</code> is also empty, so lookup continues in its enclosing environment, which is <code>enva</code>, and now lookup results in an infinite loop.</p> + +<h3 id="an-intuition-for-scoping-in-r">An intuition for scoping in R</h3> + +<p>Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1"># example 1</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> + +<span class="c1"># example 2</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>It seems that <code>x</code> takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What&rsquo;s missing from our intuition?</p> + +<p>The key insight is that R is <em>function scoped</em>. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <code>&lt;-</code> <em>does not create a new scope</em>, and it is more useful to think of it as a mutation <em>on the current environment</em>. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)</p> + +<p>In a sense, it might be more accurate to say that R <em>environments</em> are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.</p> + +<h2 id="discussion">Discussion</h2> + +<p>All of this might make you a little uncomfortable, and uncertain about R&rsquo;s scoping rules.</p> + +<p>On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply &ldquo;lexical-like&rdquo; behaviour, and the creators have confirmed that lexical scoping was the intent.</p> + +<p>On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment&mdash;but this is not consistent with R&rsquo;s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>This ambiguity shows up in a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-3-definition" name="2019-09-10-scoping-in-r-footnote-3-return">3</a></sup> where the authors write:</p> + +<blockquote> + <p>Furthermore, because variable scoping in R is dynamic and can be modified at the language level [&hellip;] it cannot be trivially guaranteed that <code>x</code> is going to point to the same data structure throughout the entire execution of the loop.</p></blockquote> + +<p>It is true that a variable <code>x</code> may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.</p> + +<p>It is true that variable <em>lookup</em> is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of <em>dynamic scope</em>, then it would be fair to say that R is dynamically scoped.</p> + +<p>But if your definition of <em>dynamic scope</em> is &ldquo;a variable is bound to the most recent assignment during the program&rsquo;s execution,&rdquo; then it is not correct to say R is dynamically scoped.</p> + +<p>I think we have this ambiguity because <em>scope</em> (the places in a program where a variable can be referenced) and <em>variable lookup</em> or <em>name resolution</em> (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.</p> + +<p>Personally, I prefer the definition of <em>scope</em> that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is <em>both</em> lexically and dynamically scoped, or that R is <em>neither</em> lexically and dynamically scoped.</p> + +<p>I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language&mdash;when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.</p> + +<p>Regardless, it is very easy to miscommunicate, so I think it&rsquo;s important to be very clear and make sure you and your audience know what definitions of scoping you&rsquo;re using!</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>This entire adventure started when we were working on a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-4-definition" name="2019-09-10-scoping-in-r-footnote-4-return">4</a></sup> and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!</p> + +<p>This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.</p> + +<p>To summarize, I would say that <em>scope</em> refers to the places in a program where a variable is visible and can be referenced. Under <em>lexical scoping</em>, the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program. Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment during the program&rsquo;s execution.</p> + +<p>I would say that R <em>aims</em> to be lexically scoped&mdash;it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!</p> + +<p>If you want more content about R and scoping, the <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">third and final part</a> of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.</p> + +<p><strong>Edited 2020/02/21:</strong> For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), <a href="http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/">this blog post</a> has some nice examples and diagrams.</p> + +<p><em>I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.</em></p> + +<p><em>If you liked this post, you may also be interested in the following Twitter threads about R: <a href="https://twitter.com/mhyee/status/1063983175163158531">one</a>, <a href="https://twitter.com/mhyee/status/1067818720532316166">two</a> and <a href="https://twitter.com/mhyee/status/1074744049951739905">three</a>.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>F. Morandat, B. Hill, L. Osvald, J. Vitek. &ldquo;Evaluating the Design of the R Language,&rdquo; in <em>Proceedings of the European Conference on Object-Oriented Programming (ECOOP)</em>, 2012. [<a href="https://doi.org/10.1007/978-3-642-31057-7_6">DOI</a>][<a href="http://janvitek.org/pubs/ecoop12.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. &ldquo;Lexical Scope and Statistical Computing&rdquo;, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-2-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-3-definition" class="footnote-definition"> + <p>L. Stadler, A. Welc, C. Humer, and M. Jordan. &ldquo;Optimizing R Language Execution via Aggressive Speculation,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2016. [<a href="https://doi.org/10.1145/2989225.2989236">DOI</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-3-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-4-definition" class="footnote-definition"> + <p>O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. &ldquo;R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2019. To appear. [<a href="http://janvitek.org/pubs/dls19.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-4-return">↩</a></p></li></ol></div> + + Lexical and Dynamic Scope + + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + 2019-09-05T10:00:00Z + 2019-09-05T10:00:00Z + + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> + + On-Stack Replacement + + urn:http-prl-ccs-neu-edu:-blog-2019-01-28-on-stack-replacement + 2019-01-28T10:29:57Z + 2019-01-28T10:29:57Z + + Ming-Ho Yee + +<p>Last semester, I took <a href="https://course.ccs.neu.edu/cs7600/">a course</a> where the final project was to write a survey paper on &ldquo;a topic in the intersection between computer systems and your area.&rdquo; So I wrote about on-stack replacement.</p> +<!-- more--> + +<p><strong>Abstract</strong></p> + +<blockquote> + <p>On-stack replacement (OSR) is a programming language implementation technique that allows a running program to switch to a different version of code. For example, a program could start executing optimized code, and then transfer to and start executing unoptimized code. This was the original use case for OSR, to facilitate debugging of optimized code.</p> + <p>After its original use was established, OSR shifted to a different use case: optimizing programs. OSR allows the run-time system to detect if a program is executing an inefficient loop, recompile and optimize the method that contains the loop, and then transfer control to the newly compiled method. Another strategy is to optimize code based on some assumptions, then, if the assumptions are invalidated at run-time, transfer control back to the original, unoptimized code.</p> + <p>In this survey paper, we study how OSR was first introduced as a means for debugging, how it came to be used for program optimizations, its implementation as a reusable library, and other directions of research.</p></blockquote> + +<p>If you&rsquo;re interested, you can find a copy <a href="/img/cs7600-mhyee-survey-paper-osr.pdf">here</a> or on <a href="https://www.overleaf.com/read/smcmsnksxfdk">Overleaf</a>.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in <a href="http://prl.ccs.neu.edu/blog/2017/03/15/tracing-jits-for-dynamic-languages/">tracing JITs for dynamic languages</a>.</em></p> + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Report: PLISS 2017 + + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-report-pliss-2017 + 2017-06-05T15:47:59Z + 2017-06-05T15:47:59Z + + Ming-Ho Yee + +<p>Two weeks ago, I attended the first <a href="https://pliss2017.github.io/">Programming Language Implementation Summer School</a>, held in beautiful Bertinoro, Italy.</p> + +<p>The goal of PLISS was &ldquo;to prepare early graduate students and advanced undergraduates for research in the field,&rdquo; and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.</p> +<!-- more--> + +<h2 id="the-talks">The Talks</h2> + +<p>With the amount of technical content at PLISS, there was easily something for everyone. <a href="http://janvitek.org/">Jan Vitek</a> and <a href="http://tratt.net/laurie/">Laurence Tratt</a> gave lectures that included hands-on exercises where we worked on JITs. <a href="https://www.cs.purdue.edu/homes/suresh/">Suresh Jagannathan</a> dived into the operational semantics of a distributed system, so we could reason about different weak consistency models. Francesco Logozzo gave us a whirlwind tour of abstract interpretation.</p> + +<p>Most of my favorite talks included some form of extra content, such as exercises, live-coding presentations, or demos. I found it really helpful to write actual code and apply what I had just learned, or to look at some concrete examples. The examples and exercises also helped with the pacing, as actively listening to four 90-minute talks every day is exhausting!</p> + +<p>Off the top of my head, these were some of my favorite talks:</p> + +<ul> + <li> + <p><strong>Dynamic Programming Language Implementation with LLVM</strong>, by Petr Maj, Oli Flückiger, and <a href="http://janvitek.org/">Jan Vitek</a>. As the first talk of the summer school, this was a gentle introduction for the rest of the week. We had <a href="https://github.com/PRL-PRG/pliss-rift/">exercises</a> (with intentional bugs to make us think!), and also brief overviews of intermediate languages, static analysis, and garbage collection. These three topics would later show up in more detail.</p></li> + <li> + <p><strong>Micro Virtual Machines</strong>, by <a href="http://users.cecs.anu.edu.au/~steveb/">Steve Blackburn</a>. This talk covered background information on virtual machines, and also the <a href="http://microvm.github.io/">Micro VM</a> project that Steve&rsquo;s group has been working on. A lot of the material was already familiar to me, but I still enjoyed the talk, and even got a few ideas for the project I&rsquo;m working on!</p></li> + <li> + <p><strong>Static Analysis</strong>, by <a href="http://matt.might.net/">Matt Might</a>. Matt&rsquo;s talk was based on one of his <a href="http://matt.might.net/articles/intro-static-analysis/">articles</a> and an older talk he&rsquo;s given. Impressively, the entire example was live-coded, with only a single mistake!</p></li> + <li> + <p><strong>Testing Language Implementations</strong>, by <a href="http://multicore.doc.ic.ac.uk/">Alastair Donaldson</a>. This was an entertaining talk, since Ally showed multiple examples of crashing compilers, and causing other kinds of mischief by triggering compiler bugs.</p></li></ul> + +<p>If you&rsquo;re disappointed that you couldn&rsquo;t see these talks, don&rsquo;t worry! The talks were recorded and will be posted very shortly.</p> + +<h2 id="the-people">The People</h2> + +<p>But there&rsquo;s more to PLISS than the talks. I&rsquo;m referring to <em>networking</em>, or the opportunity to get out and talk to other people about research.</p> + +<p>As an early graduate student, I&rsquo;ve been given a lot of advice about talking to people at conferences and the importance of the &ldquo;hallway track.&rdquo; I still have difficulty doing this at an actual conference, like <a href="http://pldi17.sigplan.org/home">PLDI</a> or <a href="http://2017.ecoop.org/">ECOOP</a>. When there are hundreds of attendees, or when people already know each other and are in conversation groups, I find it difficult to approach them.</p> + +<p>This was not the case at PLISS. There were fewer attendees: about fifty students and a dozen speakers. There was a good mix of undergraduate, master&rsquo;s, first-year PhD, and more senior PhD students. All our breakfasts, lunches, and breaks were together, so we would see the same people again and again, and inevitably start to learn each other&rsquo;s names. The speakers would also be among us, and there was a good ratio of speakers to students for discussions and mealtime mentoring.</p> + +<p>I had many opportunities to practice my &ldquo;research pitch.&rdquo; I talked to senior students and got advice. I talked to junior students and gave advice. Two different people I talked to about my research pointed me to the same paper to read. I found another student who was working with <a href="http://research.cs.wisc.edu/wpis/papers/popl95.pdf">IFDS</a>, an algorithm I have spent much time trying to understand. And, one day at lunch, my table discovered that we were all working on static analysis!</p> + +<p>As much as I enjoyed the talks, I think the best part of PLISS was meeting and talking to other people. You can replace talks with videos (but you lose the speaker-audience interaction), and you can replace conversations with other forms of communication. But there isn&rsquo;t really anything that can replace the serendipity of bumping into someone with a shared interest.</p> + +<h2 id="the-location">The Location</h2> + +<p>Actually, the <em>other</em> best part of PLISS was the location. Italy is a beautiful country with delicious food. And Bertinoro is a small town on the top of a hill, with a breathtaking view of the surrounding area. The lectures were held in a <a href="https://pliss2017.github.io/images/pics/7.jpg">castle at the top of the hill</a> (photo credit: Steve Blackburn). The speakers lived in the castle for the week, while the students lived in the former monastery (seems fitting), which has been renovated into a university residence.</p> + +<p>Here are my two favorite pictures I took (click for full size):</p> + +<p><a href="/img/pliss2017-1.jpg"><img src="/img/pliss2017-1-thumb.jpg" alt="View from the castle" /></a></p> + +<p><a href="/img/pliss2017-2.jpg"><img src="/img/pliss2017-2-thumb.jpg" alt="Panorama" /></a></p> + +<p>Steve Blackburn has more pictures posted on the <a href="https://pliss2017.github.io/">PLISS website</a>.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>PLISS was a wonderful event. Many thanks need to be given to the speakers, organizers, and sponsors, for making this possible!</p> + +<p>If and when there is a second PLISS, I highly encourage students to apply! You will learn a lot from the lectures, from talking to the speakers, and meeting other students. And if it&rsquo;s in Bertinoro again, you can enjoy the weather and nice view!</p> + + Tracing JITs for Dynamic Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-03-15-tracing-jits-for-dynamic-languages + 2017-03-15T10:54:39Z + 2017-03-15T10:54:39Z + + Ming-Ho Yee + <!-- more--> + +<p>Traditional JIT (just-in-time) compilers are method-based: they compile &ldquo;hot&rdquo; (i.e. frequently executed) methods to native code. An alternative is trace-based or tracing JITs, where the compilation unit is a (hot) sequence of instructions. Typically, such sequences of instructions correspond to loops, where programs spend most of their execution time.</p> + +<p>Where did the idea of tracing come from? What was appealing about it? How was tracing adapted for JITs and dynamic languages? What happened to Mozilla&rsquo;s TraceMonkey, which used to be part of Firefox? Do any JITs today use tracing?</p> + +<p>In this talk, I trace tracing JITs from their origins to some of their recent developments. I cover five papers: the original tracing paper, an implementation of a tracing JIT for Java, the TraceMonkey JIT for JavaScript, PyPy&rsquo;s &ldquo;meta-level&rdquo; tracing, and a specific class of optimizations for tracing JITs.</p> + +<p><em>(The idea of using the phrase &ldquo;trace tracing JITs&rdquo; is from Matthias Felleisen.)</em></p> + +<p>All materials can be found in the <a href="https://github.com/nuprl/hopl-s2017/tree/master/tracing-jit">course repository</a>:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/tracing-jit/notes.pdf">Full notes</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/tracing-jit/annotated.txt">Annotated bibliography</a></li></ul> + +<hr /> + +<p><em>If you liked this post, you may also be interested in <a href="http://prl.ccs.neu.edu/blog/2019/01/28/on-stack-replacement/">on-stack replacement</a>.</em></p> + + Fall 2016 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + 2017-01-02T16:39:37Z + 2017-01-02T16:39:37Z + Ben Chung + Milo Davis + Ming-Ho Yee + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Ming-Ho-Yee.rss.xml b/blog/feeds/Author-Ming-Ho-Yee.rss.xml new file mode 100644 index 00000000..6e0b8452 --- /dev/null +++ b/blog/feeds/Author-Ming-Ho-Yee.rss.xml @@ -0,0 +1,1216 @@ + + + + PRL Blog: Posts tagged 'Author: Ming-Ho Yee' + PRL Blog: Posts tagged 'Author: Ming-Ho Yee' + http://prl.ccs.neu.edu/blog/tags/Author-Ming-Ho-Yee.html + Tue, 10 Sep 2019 11:00:00 UT + Tue, 10 Sep 2019 11:00:00 UT + 1800 + + Four Kinds of Scoping in R + http://prl.ccs.neu.edu/blog/2019/09/10/four-kinds-of-scoping-in-r/?utm_source=Author-Ming-Ho-Yee&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-four-kinds-of-scoping-in-r + Tue, 10 Sep 2019 11:00:00 UT + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">first</a> and <a href="/blog/2019/09/10/scoping-in-r/">second</a> parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.</p> + +<p>In this third and final part of my blog series, I&rsquo;d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.</p> + +<p>This is a &ldquo;bonus&rdquo; blog post, because I&rsquo;m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.</p> +<!-- more--> + +<h2 id="lexical-scope-and-statistical-computation">Lexical scope and statistical computation</h2> + +<p>In <em>Lexical Scope and Statistical Computation</em>,<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">1</a></sup> Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.</p> + +<p>For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:</p> + +<ul> + <li><em>trivial</em>: no free variables allowed</li> + <li><em>static</em>: a free variable takes its value from a set of global variables</li> + <li><em>lexical</em>: a free variable takes the value of the binding that was in effect when the function was defined</li> + <li><em>dynamic</em>: a free variable takes the value of the most recent assignment to that variable</li></ul> + +<p>Note that under this set of definitions, <em>static scoping</em> is a separate scoping rule and not another name for <em>lexical scoping</em>.</p> + +<p>It is possible to simulate each of strategies in R. For fun, we can even construct &ldquo;factories&rdquo; that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)</p> + +<h3 id="template">Template</h3> + +<p>Our examples will follow the template given below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">factory</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="o">&lt;???&gt;</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">factory</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># error, 0, 1, or 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We want to define a <code>factory</code> that takes a function literal and returns a closure that implements the desired scoping rule.</p> + +<p>Our example consists of three definitions of <code>x</code>. On line 5, we assign <code>0</code> to <code>x</code> at the top level. On line 7, we assign <code>1</code> to <code>x</code> inside function <code>h</code>, where we also create the closure. On line 12, we assign <code>2</code> to <code>x</code> inside the function <code>f</code> and right before we call <code>g</code>, which is the closure.</p> + +<p>Finally, we call <code>f</code> and observe the result:</p> + +<ul> + <li>Under trivial scoping, no free variables are allowed, so <code>f()</code> should result in an error.</li> + <li>Under static scoping, free variables may only refer to global variables, so <code>f()</code> should return <code>0</code>.</li> + <li>Under lexical scoping, free variables refer to the variables in scope when the function was defined, so <code>f()</code> should return <code>1</code>.</li> + <li>Under dynamic scoping, free variables take the value from the most recent assignment, so <code>f()</code> should return <code>2</code>.</li></ul> + +<p>We will implement the body of <code>factory</code> in only 3&ndash;5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to <code>factory</code> on line 10.</p> + +<h3 id="trivial-scoping">Trivial scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeTrivial</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">baseenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeTrivial</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># Error in f(0) : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>substitute</code> returns the unevaluated parse tree for <code>fun</code>. In other words, it obtains the literal argument that was passed for <code>fun</code>. This works because of call-by-need semantics in R: function arguments are packaged up into <em>promises</em>. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">2</a></sup> discusses laziness in R in more detail.</p> + +<p>In this example, on line 8, we call <code>factory</code> with <code>function(a) x+a</code> as the argument for the formal parameter <code>fun</code>. Then, we evaluate that parse tree with <code>eval</code>.</p> + +<p>At this point, <code>res</code> is the closure with expression <code>function(a) x+a</code> and a reference to the environment of <code>makeTrivial</code>. On line 3, we change that reference to <code>baseenv()</code>, which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.</p> + +<p>Therefore, variable lookup in the function literal will only search the base environment, so <code>f()</code> results in an error.</p> + +<h3 id="static-scoping">Static scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeStatic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">globalenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeStatic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, on line 3, we update the environment of <code>res</code> to refer to <code>globalenv()</code>, which is the top-level environment where globals are defined.</p> + +<p>Therefore, variable lookup searches the top-level environment, so <code>f()</code> returns <code>0</code>.</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeLexical</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeLexical</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of <code>res</code> to <code>parent.frame()</code>, which is the environment of the function (<code>h</code>) that called the current function (<code>makeLexical</code>). This allows us to simulate lexical scoping, as if the function literal was evaluated inside <code>h</code>, rather than <code>makeLexical</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>h</code>, so <code>f()</code> returns <code>1</code>.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeDynamic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">function</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="nf">res</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> + <span class="p">}</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeDynamic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, we need another level of indirection. <code>makeDynamic</code> returns an anonymous function literal. The anonymous function takes <code>...</code>, which represents an arbitrary list of arguments, and then on line 5 we call <code>res</code> with those exact arguments. Note that we set the environment of <code>res</code> to be the environment of the <em>caller</em> of the anonymous function. Because of the multiple levels of indirection, the caller is <code>f</code>, on line 17.</p> + +<p>On line 12, <code>makeDynamic</code> returns a closure for the anonymous function. <code>h</code> returns that closure when it is called, and assigns it to <code>g</code>. When <code>g</code> is called on line 17, the function literal <code>function(a) x+a</code> is finally evaluated, and its environment is set to the environment of <code>f</code>, the caller of <code>g</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>f</code>, so <code>f()</code> returns <code>2</code>.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the <a href="/blog/2019/09/10/scoping-in-r/">previous post</a>, it&rsquo;s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.</p> + +<p>And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.</p> + +<p><em>I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>A. Goel and J. Vitek. &ldquo;On the Design, Implementation and Use of Laziness in R,&rdquo; in <em>Proceedings of the ACM in Programming Languages (PACMPL)</em>, vol. 3, no. OOPSLA, 2019. To appear. [<a href="http://janvitek.org/pubs/oopsla19a.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">↩</a></p></li></ol></div> + + Scoping in R + http://prl.ccs.neu.edu/blog/2019/09/10/scoping-in-r/?utm_source=Author-Ming-Ho-Yee&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-scoping-in-r + Tue, 10 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">previous post</a> of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: <em>is R lexically or dynamically scoped?</em></p> +<!-- more--> + +<p>Recall the example program from before:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s examine what happens when we run this example. First, we create a mapping for <code>x</code> in the top-level environment. On line 2, we define a function <code>f</code>, which returns the value of some <code>x</code>. On line 3, we define a function <code>g</code>, which creates a new mapping for <code>x</code>, and then calls <code>f</code>. Note that the assignment on line 4 does <em>not</em> update the definition on line 1.</p> + +<p>When <code>f</code> is called, it needs to look up the value of <code>x</code>. In other words, does the reference of <code>x</code> on line 2 refer to the assignment on line 1 or the assignment on line 4? If <code>f</code> returns <code>1</code>, then the behaviour matches lexical scoping. If it returns <code>2</code>, then the behaviour matches dynamic scoping.</p> + +<p>When we run this example, the result is <code>1</code>. This implies that R is lexically scoped.</p> + +<p>But there&rsquo;s more to this story. In the rest of this blog post, I&rsquo;ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.</p> + +<p>The <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">next and final part</a> of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.</p> + +<h2 id="r-is-lexically-scoped-but">R is lexically scoped, but&hellip;</h2> + +<p>In <em>Evaluating the Design of the R Language</em>,<sup><a href="#2019-09-10-scoping-in-r-footnote-1-definition" name="2019-09-10-scoping-in-r-footnote-1-return">1</a></sup> Morandat, Hill, Osvald, and Vitek write:</p> + +<blockquote> + <p>As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.</p></blockquote> + +<p>In other words, R provides many different &ldquo;escape hatches&rdquo;&mdash;ways to bypass lexical scoping. Additionally, even without escape hatches, some of R&rsquo;s functionality can be surprising.</p> + +<h3 id="functions-environments-and-variables-in-r">Functions, environments, and variables in R</h3> + +<p>Before we look at some examples, I think it&rsquo;s useful to briefly discuss some of the core concepts in R that relate to scoping.</p> + +<ul> + <li> + <p><strong>Functions.</strong> R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.</p></li> + <li> + <p><strong>Environments.</strong> An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the &ldquo;enclosing&rdquo; environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.</p></li> + <li> + <p><strong>Variable lookup.</strong> When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.</p></li> + <li> + <p><strong>Variable assignment.</strong> <code>&lt;-</code> is the variable assignment operator in R. The expression <code>x &lt;- 1</code> assigns the value <code>1</code> to the variable <code>x</code> in the current environment. If a mapping for <code>x</code> already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.</p></li></ul> + +<p>From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.<sup><a href="#2019-09-10-scoping-in-r-footnote-2-definition" name="2019-09-10-scoping-in-r-footnote-2-return">2</a></sup></p> + +<p>On the other hand, variable lookup depends on the run-time state of the program&mdash;names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.</p> + +<p>The following examples will make this clear.</p> + +<h3 id="examples">Examples</h3> + +<h4 id="adding-variable-mappings-at-run-time">Adding variable mappings at run time</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it creates a function <code>g</code> that returns <code>x</code>, assigns <code>2</code> to <code>x</code>, and then calls <code>g</code>. When <code>g</code> is called, it looks up <code>x</code>. Since no mapping is found in <code>g</code>&rsquo;s environment, it searches in the enclosing environment (<code>f</code>&rsquo;s), and finds that <code>x</code> has value <code>2</code>. Therefore, <code>g</code> returns <code>2</code>.</p> + +<p>Note that the <code>x</code> on line 3 is resolved only when function <code>g</code> is called, not when it is defined. However, when <code>g</code> is defined, its environment has a reference to <code>f</code>&rsquo;s environment. Therefore, as long as <code>x</code> is defined <em>before</em> <code>g</code> is called, the lookup will always succeed.</p> + +<p>Here&rsquo;s a second example:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>f</code> is a function that branches on its argument, <code>b</code>. If <code>b</code> evaluates to true, then the expression <code>x &lt;- 2</code> is evaluated, and a mapping for <code>x</code> is created in <code>f</code>&rsquo;s environment. Otherwise, no mapping is created.</p> + +<p>When we look up the value of <code>x</code> on line 5, R will first search the function&rsquo;s environment. If <code>b</code> evaluated to true, then R will find a value for <code>x</code>, which is <code>2</code>. Otherwise, R will search in the enclosing environment of <code>f</code>, and find that <code>x</code> is <code>1</code>.</p> + +<p>Both of these examples vaguely resemble dynamic scoping, in that <code>x</code> takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.</p> + +<h4 id="function-lookup">Function lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> <span class="c1"># not an error</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">(</span><span class="m">42</span><span class="p">)</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.</p> + +<p>In this example, we call <code>g</code> with the argument <code>42</code>, which is not a function. Then, in the body of <code>g</code>, we call <code>f(0)</code> on line 3, which requires looking up <code>f</code>. Although there is an <code>f</code> in the environment of <code>g</code>, its value is <code>42</code>, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so <code>f(0)</code> returns <code>0</code>.</p> + +<p>This behaviour exists because <code>c</code> is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.</p> + +<h4 id="super-assignment">Super assignment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="n">x</span> <span class="o">&lt;&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +<span class="n">x</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>&lt;&lt;-</code> is the &ldquo;super assignment&rdquo; operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.</p> + +<p>In the above program, we define <code>x</code> to be <code>0</code> at the top level, and then define the function <code>f</code>. When we call <code>f</code> on line 7, it assigns <code>1</code> to <code>x</code> on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, <code>f</code> returns <code>x</code>, which is looked up from the local environment and has value <code>1</code>. Finally, line 8 looks up <code>x</code> from the top level environment, which has value <code>2</code>.</p> + +<h4 id="evaluating-arbitrary-code">Evaluating arbitrary code</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">eval</span><span class="p">(</span><span class="nf">parse</span><span class="p">(</span><span class="n">text</span> <span class="o">=</span> <span class="n">t</span><span class="p">))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="s">"x &lt;- 0"</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument <code>t</code>, which happens to be the string <code>"x &lt;- 0"</code>. Then, when line 4 executes, the lookup of <code>x</code> returns <code>0</code>.</p> + +<h4 id="simulating-dynamic-scope">Simulating dynamic scope</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.frame</span><span class="p">())</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 3, we perform an explicit variable lookup for <code>x</code>, but we do so in the environment <code>parent.frame()</code>, which refers to the calling function&rsquo;s environment, in this case, <code>g</code>&rsquo;s environment.. Therefore, the lookup returns <code>2</code>.</p> + +<p>Note that R has a similarly named function, <code>parent.env(e)</code> which returns the <em>enclosing environment</em> of the given environment <code>e</code>.</p> + +<h4 id="constructing-an-arbitrary-environment">Constructing an arbitrary environment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">e</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> + <span class="n">e</span><span class="o">$</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">3</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="n">e</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 3</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it constructs a new environment, <code>e</code>, which is initially empty. (By default, its enclosing environment is the current environment, which is <code>f</code>&rsquo;s.) Next, on line 4, it directly adds a mapping to that environment, assigning <code>3</code> to <code>x</code>. Then, on line 5, the lookup is explicitly done in environment <code>e</code>, so <code>f</code> returns <code>3</code>.</p> + +<h4 id="deleting-mappings">Deleting mappings</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">rm</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.env</span><span class="p">(</span><span class="nf">environment</span><span class="p">()))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># Error in f() : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Not only is it possible to dynamically add and modify mappings in R, but it is also possible to <em>delete</em> mappings. This is what line 3 does: it explicitly removes the mapping for <code>x</code> from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when <code>f</code> is called, the lookup of <code>x</code> fails and an error is raised.</p> + +<h4 id="infinite-loop-during-variable-lookup">Infinite loop during variable lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">enva</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="n">envb</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">enva</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">envb</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">envb</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="nf">environment</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="nf">f</span><span class="p">()</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.</p> + +<p>On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: <code>enva</code>&rsquo;s enclosing environment is <code>envb</code>, and <code>envb</code>&rsquo;s enclosing environment is <code>enva</code>.</p> + +<p>On line 5, we define a function with a free variable, <code>x</code>, but on line 6, we set <code>f</code>&rsquo;s environment to be <code>enva</code>. Finally, we call <code>f</code>.</p> + +<p>When the body of <code>f</code> is evaluated, it needs to look up <code>x</code>. Lookup starts in <code>f</code>&rsquo;s environment, which we set to be <code>enva</code>. Since no mapping for <code>x</code> is found, lookup continues in <code>enva</code>&rsquo;s enclosing environment, which is <code>envb</code>. However, <code>envb</code> is also empty, so lookup continues in its enclosing environment, which is <code>enva</code>, and now lookup results in an infinite loop.</p> + +<h3 id="an-intuition-for-scoping-in-r">An intuition for scoping in R</h3> + +<p>Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1"># example 1</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> + +<span class="c1"># example 2</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>It seems that <code>x</code> takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What&rsquo;s missing from our intuition?</p> + +<p>The key insight is that R is <em>function scoped</em>. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <code>&lt;-</code> <em>does not create a new scope</em>, and it is more useful to think of it as a mutation <em>on the current environment</em>. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)</p> + +<p>In a sense, it might be more accurate to say that R <em>environments</em> are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.</p> + +<h2 id="discussion">Discussion</h2> + +<p>All of this might make you a little uncomfortable, and uncertain about R&rsquo;s scoping rules.</p> + +<p>On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply &ldquo;lexical-like&rdquo; behaviour, and the creators have confirmed that lexical scoping was the intent.</p> + +<p>On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment&mdash;but this is not consistent with R&rsquo;s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>This ambiguity shows up in a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-3-definition" name="2019-09-10-scoping-in-r-footnote-3-return">3</a></sup> where the authors write:</p> + +<blockquote> + <p>Furthermore, because variable scoping in R is dynamic and can be modified at the language level [&hellip;] it cannot be trivially guaranteed that <code>x</code> is going to point to the same data structure throughout the entire execution of the loop.</p></blockquote> + +<p>It is true that a variable <code>x</code> may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.</p> + +<p>It is true that variable <em>lookup</em> is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of <em>dynamic scope</em>, then it would be fair to say that R is dynamically scoped.</p> + +<p>But if your definition of <em>dynamic scope</em> is &ldquo;a variable is bound to the most recent assignment during the program&rsquo;s execution,&rdquo; then it is not correct to say R is dynamically scoped.</p> + +<p>I think we have this ambiguity because <em>scope</em> (the places in a program where a variable can be referenced) and <em>variable lookup</em> or <em>name resolution</em> (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.</p> + +<p>Personally, I prefer the definition of <em>scope</em> that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is <em>both</em> lexically and dynamically scoped, or that R is <em>neither</em> lexically and dynamically scoped.</p> + +<p>I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language&mdash;when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.</p> + +<p>Regardless, it is very easy to miscommunicate, so I think it&rsquo;s important to be very clear and make sure you and your audience know what definitions of scoping you&rsquo;re using!</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>This entire adventure started when we were working on a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-4-definition" name="2019-09-10-scoping-in-r-footnote-4-return">4</a></sup> and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!</p> + +<p>This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.</p> + +<p>To summarize, I would say that <em>scope</em> refers to the places in a program where a variable is visible and can be referenced. Under <em>lexical scoping</em>, the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program. Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment during the program&rsquo;s execution.</p> + +<p>I would say that R <em>aims</em> to be lexically scoped&mdash;it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!</p> + +<p>If you want more content about R and scoping, the <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">third and final part</a> of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.</p> + +<p><strong>Edited 2020/02/21:</strong> For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), <a href="http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/">this blog post</a> has some nice examples and diagrams.</p> + +<p><em>I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.</em></p> + +<p><em>If you liked this post, you may also be interested in the following Twitter threads about R: <a href="https://twitter.com/mhyee/status/1063983175163158531">one</a>, <a href="https://twitter.com/mhyee/status/1067818720532316166">two</a> and <a href="https://twitter.com/mhyee/status/1074744049951739905">three</a>.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>F. Morandat, B. Hill, L. Osvald, J. Vitek. &ldquo;Evaluating the Design of the R Language,&rdquo; in <em>Proceedings of the European Conference on Object-Oriented Programming (ECOOP)</em>, 2012. [<a href="https://doi.org/10.1007/978-3-642-31057-7_6">DOI</a>][<a href="http://janvitek.org/pubs/ecoop12.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. &ldquo;Lexical Scope and Statistical Computing&rdquo;, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-2-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-3-definition" class="footnote-definition"> + <p>L. Stadler, A. Welc, C. Humer, and M. Jordan. &ldquo;Optimizing R Language Execution via Aggressive Speculation,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2016. [<a href="https://doi.org/10.1145/2989225.2989236">DOI</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-3-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-4-definition" class="footnote-definition"> + <p>O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. &ldquo;R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2019. To appear. [<a href="http://janvitek.org/pubs/dls19.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-4-return">↩</a></p></li></ol></div> + + Lexical and Dynamic Scope + http://prl.ccs.neu.edu/blog/2019/09/05/lexical-and-dynamic-scope/?utm_source=Author-Ming-Ho-Yee&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + Thu, 05 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> + + On-Stack Replacement + http://prl.ccs.neu.edu/blog/2019/01/28/on-stack-replacement/?utm_source=Author-Ming-Ho-Yee&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-01-28-on-stack-replacement + Mon, 28 Jan 2019 10:29:57 UT + Ming-Ho Yee + +<p>Last semester, I took <a href="https://course.ccs.neu.edu/cs7600/">a course</a> where the final project was to write a survey paper on &ldquo;a topic in the intersection between computer systems and your area.&rdquo; So I wrote about on-stack replacement.</p> +<!-- more--> + +<p><strong>Abstract</strong></p> + +<blockquote> + <p>On-stack replacement (OSR) is a programming language implementation technique that allows a running program to switch to a different version of code. For example, a program could start executing optimized code, and then transfer to and start executing unoptimized code. This was the original use case for OSR, to facilitate debugging of optimized code.</p> + <p>After its original use was established, OSR shifted to a different use case: optimizing programs. OSR allows the run-time system to detect if a program is executing an inefficient loop, recompile and optimize the method that contains the loop, and then transfer control to the newly compiled method. Another strategy is to optimize code based on some assumptions, then, if the assumptions are invalidated at run-time, transfer control back to the original, unoptimized code.</p> + <p>In this survey paper, we study how OSR was first introduced as a means for debugging, how it came to be used for program optimizations, its implementation as a reusable library, and other directions of research.</p></blockquote> + +<p>If you&rsquo;re interested, you can find a copy <a href="/img/cs7600-mhyee-survey-paper-osr.pdf">here</a> or on <a href="https://www.overleaf.com/read/smcmsnksxfdk">Overleaf</a>.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in <a href="http://prl.ccs.neu.edu/blog/2017/03/15/tracing-jits-for-dynamic-languages/">tracing JITs for dynamic languages</a>.</em></p> + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=Author-Ming-Ho-Yee&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Report: PLISS 2017 + http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/?utm_source=Author-Ming-Ho-Yee&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-report-pliss-2017 + Mon, 05 Jun 2017 15:47:59 UT + Ming-Ho Yee + +<p>Two weeks ago, I attended the first <a href="https://pliss2017.github.io/">Programming Language Implementation Summer School</a>, held in beautiful Bertinoro, Italy.</p> + +<p>The goal of PLISS was &ldquo;to prepare early graduate students and advanced undergraduates for research in the field,&rdquo; and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.</p> +<!-- more--> + +<h2 id="the-talks">The Talks</h2> + +<p>With the amount of technical content at PLISS, there was easily something for everyone. <a href="http://janvitek.org/">Jan Vitek</a> and <a href="http://tratt.net/laurie/">Laurence Tratt</a> gave lectures that included hands-on exercises where we worked on JITs. <a href="https://www.cs.purdue.edu/homes/suresh/">Suresh Jagannathan</a> dived into the operational semantics of a distributed system, so we could reason about different weak consistency models. Francesco Logozzo gave us a whirlwind tour of abstract interpretation.</p> + +<p>Most of my favorite talks included some form of extra content, such as exercises, live-coding presentations, or demos. I found it really helpful to write actual code and apply what I had just learned, or to look at some concrete examples. The examples and exercises also helped with the pacing, as actively listening to four 90-minute talks every day is exhausting!</p> + +<p>Off the top of my head, these were some of my favorite talks:</p> + +<ul> + <li> + <p><strong>Dynamic Programming Language Implementation with LLVM</strong>, by Petr Maj, Oli Flückiger, and <a href="http://janvitek.org/">Jan Vitek</a>. As the first talk of the summer school, this was a gentle introduction for the rest of the week. We had <a href="https://github.com/PRL-PRG/pliss-rift/">exercises</a> (with intentional bugs to make us think!), and also brief overviews of intermediate languages, static analysis, and garbage collection. These three topics would later show up in more detail.</p></li> + <li> + <p><strong>Micro Virtual Machines</strong>, by <a href="http://users.cecs.anu.edu.au/~steveb/">Steve Blackburn</a>. This talk covered background information on virtual machines, and also the <a href="http://microvm.github.io/">Micro VM</a> project that Steve&rsquo;s group has been working on. A lot of the material was already familiar to me, but I still enjoyed the talk, and even got a few ideas for the project I&rsquo;m working on!</p></li> + <li> + <p><strong>Static Analysis</strong>, by <a href="http://matt.might.net/">Matt Might</a>. Matt&rsquo;s talk was based on one of his <a href="http://matt.might.net/articles/intro-static-analysis/">articles</a> and an older talk he&rsquo;s given. Impressively, the entire example was live-coded, with only a single mistake!</p></li> + <li> + <p><strong>Testing Language Implementations</strong>, by <a href="http://multicore.doc.ic.ac.uk/">Alastair Donaldson</a>. This was an entertaining talk, since Ally showed multiple examples of crashing compilers, and causing other kinds of mischief by triggering compiler bugs.</p></li></ul> + +<p>If you&rsquo;re disappointed that you couldn&rsquo;t see these talks, don&rsquo;t worry! The talks were recorded and will be posted very shortly.</p> + +<h2 id="the-people">The People</h2> + +<p>But there&rsquo;s more to PLISS than the talks. I&rsquo;m referring to <em>networking</em>, or the opportunity to get out and talk to other people about research.</p> + +<p>As an early graduate student, I&rsquo;ve been given a lot of advice about talking to people at conferences and the importance of the &ldquo;hallway track.&rdquo; I still have difficulty doing this at an actual conference, like <a href="http://pldi17.sigplan.org/home">PLDI</a> or <a href="http://2017.ecoop.org/">ECOOP</a>. When there are hundreds of attendees, or when people already know each other and are in conversation groups, I find it difficult to approach them.</p> + +<p>This was not the case at PLISS. There were fewer attendees: about fifty students and a dozen speakers. There was a good mix of undergraduate, master&rsquo;s, first-year PhD, and more senior PhD students. All our breakfasts, lunches, and breaks were together, so we would see the same people again and again, and inevitably start to learn each other&rsquo;s names. The speakers would also be among us, and there was a good ratio of speakers to students for discussions and mealtime mentoring.</p> + +<p>I had many opportunities to practice my &ldquo;research pitch.&rdquo; I talked to senior students and got advice. I talked to junior students and gave advice. Two different people I talked to about my research pointed me to the same paper to read. I found another student who was working with <a href="http://research.cs.wisc.edu/wpis/papers/popl95.pdf">IFDS</a>, an algorithm I have spent much time trying to understand. And, one day at lunch, my table discovered that we were all working on static analysis!</p> + +<p>As much as I enjoyed the talks, I think the best part of PLISS was meeting and talking to other people. You can replace talks with videos (but you lose the speaker-audience interaction), and you can replace conversations with other forms of communication. But there isn&rsquo;t really anything that can replace the serendipity of bumping into someone with a shared interest.</p> + +<h2 id="the-location">The Location</h2> + +<p>Actually, the <em>other</em> best part of PLISS was the location. Italy is a beautiful country with delicious food. And Bertinoro is a small town on the top of a hill, with a breathtaking view of the surrounding area. The lectures were held in a <a href="https://pliss2017.github.io/images/pics/7.jpg">castle at the top of the hill</a> (photo credit: Steve Blackburn). The speakers lived in the castle for the week, while the students lived in the former monastery (seems fitting), which has been renovated into a university residence.</p> + +<p>Here are my two favorite pictures I took (click for full size):</p> + +<p><a href="/img/pliss2017-1.jpg"><img src="/img/pliss2017-1-thumb.jpg" alt="View from the castle" /></a></p> + +<p><a href="/img/pliss2017-2.jpg"><img src="/img/pliss2017-2-thumb.jpg" alt="Panorama" /></a></p> + +<p>Steve Blackburn has more pictures posted on the <a href="https://pliss2017.github.io/">PLISS website</a>.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>PLISS was a wonderful event. Many thanks need to be given to the speakers, organizers, and sponsors, for making this possible!</p> + +<p>If and when there is a second PLISS, I highly encourage students to apply! You will learn a lot from the lectures, from talking to the speakers, and meeting other students. And if it&rsquo;s in Bertinoro again, you can enjoy the weather and nice view!</p> + + Tracing JITs for Dynamic Languages + http://prl.ccs.neu.edu/blog/2017/03/15/tracing-jits-for-dynamic-languages/?utm_source=Author-Ming-Ho-Yee&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-03-15-tracing-jits-for-dynamic-languages + Wed, 15 Mar 2017 10:54:39 UT + Ming-Ho Yee + <!-- more--> + +<p>Traditional JIT (just-in-time) compilers are method-based: they compile &ldquo;hot&rdquo; (i.e. frequently executed) methods to native code. An alternative is trace-based or tracing JITs, where the compilation unit is a (hot) sequence of instructions. Typically, such sequences of instructions correspond to loops, where programs spend most of their execution time.</p> + +<p>Where did the idea of tracing come from? What was appealing about it? How was tracing adapted for JITs and dynamic languages? What happened to Mozilla&rsquo;s TraceMonkey, which used to be part of Firefox? Do any JITs today use tracing?</p> + +<p>In this talk, I trace tracing JITs from their origins to some of their recent developments. I cover five papers: the original tracing paper, an implementation of a tracing JIT for Java, the TraceMonkey JIT for JavaScript, PyPy&rsquo;s &ldquo;meta-level&rdquo; tracing, and a specific class of optimizations for tracing JITs.</p> + +<p><em>(The idea of using the phrase &ldquo;trace tracing JITs&rdquo; is from Matthias Felleisen.)</em></p> + +<p>All materials can be found in the <a href="https://github.com/nuprl/hopl-s2017/tree/master/tracing-jit">course repository</a>:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/tracing-jit/notes.pdf">Full notes</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/tracing-jit/annotated.txt">Annotated bibliography</a></li></ul> + +<hr /> + +<p><em>If you liked this post, you may also be interested in <a href="http://prl.ccs.neu.edu/blog/2019/01/28/on-stack-replacement/">on-stack replacement</a>.</em></p> + + Fall 2016 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/?utm_source=Author-Ming-Ho-Yee&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + Mon, 02 Jan 2017 16:39:37 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Olek-Gierczak.atom.xml b/blog/feeds/Author-Olek-Gierczak.atom.xml new file mode 100644 index 00000000..4ce5cff1 --- /dev/null +++ b/blog/feeds/Author-Olek-Gierczak.atom.xml @@ -0,0 +1,133 @@ + + + PRL Blog: Posts tagged 'Author: Olek Gierczak' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Olek-Gierczak-html + 2019-12-12T12:51:53Z + + PRL Offsite 2019 Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2019-12-12-prl-offsite-2019-retrospective + 2019-12-12T12:51:53Z + 2019-12-12T12:51:53Z + Ben Greenman + Olek Gierczak + + Ben Greenman, Olek Gierczak + +<p>On November 11th 2019, the PRL had a private offsite meeting at the <a href="https://mtwyouth.org">More Than Words bookstore</a> in downtown Boston. For future offsite organizers, this post records what happened and how.</p> +<!-- more--> + +<h2 id="early-planning-goals">Early Planning, Goals</h2> + +<p>Every Fall, the PRL holds a kickoff meeting to assign <a href="http://prl.ccs.neu.edu/contact">roles</a> for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.</p> + +<blockquote> + <p>Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the <a href="http://www.warrencenter.com">Warren Conference Center</a> is <a href="https://www.northeastern.edu/events/northeastern-owned-off-campus-venues">Northeastern University&rsquo;s only franchised</a> off-campus venue.</p></blockquote> + +<p>The <strong>motivation</strong> for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students &mdash; both to get to know the group, and to become known. They&rsquo;re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">Spring 2017 edition</a> of HOPL.</p> + +<p>Our primary <strong>goal</strong> this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.</p> + +<p>In light of the main goal, we prefer to call the event an &ldquo;offsite&rdquo; (or &ldquo;offsite meeting&rdquo;) rather than a &ldquo;retreat&rdquo; because the target was an informative day rather than a relaxing one.</p> + +<h2 id="booking-and-logistics">Booking and Logistics</h2> + +<p>Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that&rsquo;s all we had.</p> + +<h3 id="date--nov-11">Date = Nov 11</h3> + +<p>In the beginning, we chose Veterans&rsquo; Day (2019&ndash;11&ndash;11) and Northeastern <a href="https://registrar.northeastern.edu/app/uploads/2019-2020-University-Wide-Calendar-List.pdf">reading day</a> (2019&ndash;12&ndash;05) as possible dates. We ended up with Veterans&rsquo; Day.</p> + +<p>A small number of lab members were opposed to Veterans&rsquo; Day. They gave two reasons: the Fall semester is especially busy, and Veterans&rsquo; Day is a federal holiday.</p> + +<h3 id="venue--mtw">Venue = MTW</h3> + +<p>The first venue we tried was the <a href="http://www.warrencenter.com">Warren Conference Center</a> in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the <a href="https://www.khoury.northeastern.edu/people/chelsea-smith/">Khoury Events Team</a> to reach out on our behalf. They returned with an email address and <a href="/blog/static/warren-center-meetings-and-retreats.pdf">event menu</a> (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.</p> +<!-- Donald Pepple (Northeastern) made contact with Christine Barisano (Framingham)--> + +<p>Second, we contacted the <a href="https://alumni.northeastern.edu/about/alumni-center/">Northeastern University Alumni Center</a>. They were not open on Veterans&rsquo; Day 2019.</p> + +<p>Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: <a href="https://mtwyouth.org">More Than Words</a>.</p> + +<p>More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!</p> + +<blockquote> + <p>More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you&rsquo;ll see a mix of &ldquo;employees&rdquo; and &ldquo;volunteers&rdquo; helping out.</p></blockquote> + +<p>Booking was complicated, though, by the fact that Northeastern requires <a href="http://catalog.northeastern.edu/graduate/health-sciences/academic-policies-procedures/liability-insurance/">liability insurance</a> for all off-campus events. If <strong>you</strong> are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.</p> + +<p>Here are more facts about MTW:</p> + +<ul> + <li>the theater space seats 30 with lots of extra space</li> + <li>the two long tables in the reading room can seat about 20</li> + <li>the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor</li> + <li>there&rsquo;s no whiteboard, but MTW can supply an easel stand</li> + <li>much of the furniture in store is antique and for sale, so be careful using it &mdash; both to avoid damaging it, and because antiques can be oddly-shaped</li> + <li>the bookstore was closed on Veterans&rsquo; Day, but we were able to have our event and buy books</li> + <li>MTW may have fridge space to temporarily hold leftovers</li> + <li>closest T station: <a href="https://www.mbta.com/stops/place-tumnl">Tufts Medical Center</a></li></ul> + +<h3 id="keynote--none">Keynote = None</h3> + +<p>The original, ambitious plan was to invite two keynote speakers &mdash; one working in industry and one from academia &mdash; to enrich the offsite with new knowledge. And because this was the PRL&rsquo;s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)</p> + +<p>We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.</p> + +<p>Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).</p> +<!-- to future planners: we invited Kathi Fisler; she had grant-writing plans for that day and would be interested in coming to a future offsite--> + +<h3 id="food-coffee-tea">Food, Coffee, Tea</h3> + +<p><a href="https://flourbakery.com/">Flour Bakery + Cafe</a> provided lunch. For most days, you can order from Flour using an <a href="https://flourbakery.com/menu/catering/catering-information/">online form</a>. For holidays, you may need to send an email &mdash; that&rsquo;s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan &amp; gluten-free <em>everything spiced</em> salads.</p> + +<p><a href="https://www.trycuppacoffee.com/location/">Cuppacoffee</a> provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything &mdash; the order + napkins, milk, cream cheese, butter, and knives &mdash; ourselves because Cuppacoffee is very close to MTW.</p> + +<p><a href="http://www.cmartboston.com/">CMart</a> sold us water and juices. (There is also a Whole Foods near MTW.)</p> + +<p>At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour&rsquo;s <a href="https://youtu.be/kIbhckqanHI">sticky buns</a>.</p> + +<h3 id="event-program">Event Program</h3> + +<p>The following assumptions/goals constrained the schedule for the day:</p> + +<ul> + <li>give all 17 on-campus Ph.D. students a talk slot; talks must <strong>either</strong> communicate one technical idea from recent work <strong>or</strong> say what led the speaker to apply to a PL Ph.D. program</li> + <li>allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)</li> + <li>do not make the audience sit for more than 1 hour at a time</li> + <li>maximize the number and length of breaks (to encourage discussions)</li> + <li>include a relevant and fun welcome activity</li> + <li>save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions</li></ul> + +<p>We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.</p> + +<p>The activity was <em>codewalks</em>. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition/blob/master/src/main/java/com/seriouscompany/business/java/fizzbuzz/packagenamingpackage/impl/math/arithmetics/IntegerDivider.java">IntegerDivider</a> class from a silly FizzBuzz implementation and a solution to the <a href="https://en.wikipedia.org/wiki/Dutch_national_flag_problem">Dutch national flag problem</a>). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and <a href="https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/">egoless programming</a>.</p> + +<p>The talks ran conference-style. One organizer played &ldquo;session chair&rdquo; to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.</p> + +<p>For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.</p> + +<h2 id="looking-back-at-the-details">Looking Back at the Details</h2> + +<p>Reading Day in the Spring may be a good, free date for future retreats.</p> + +<p>We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.</p> + +<p>The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we&rsquo;d keep the morning breaks short &mdash; just enough to use the restroom and grab a coffee &mdash; and aim for 30-min breaks in the afternoon.</p> + +<p>Groups of three 15-minute talks worked well, but groups of four talks might be equally good.</p> + +<p>Perhaps each speaker should get the chance to pick a talk length. <a href="https://nepls.org/">NEPLS</a>, for example, allows a choice between 5-min. and 30-min. talks.</p> + +<p>The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick &ldquo;topics&rdquo; beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.</p> +<!-- to future organizers: the panel began asking faculty for a mistake in their career and wound up with a defensive flavor--> + +<h2 id="looking-back-overall">Looking Back, Overall</h2> + +<p>This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.</p> + +<p>There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time &mdash; keep the same date &amp; program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.</p> + +<p>With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Olek-Gierczak.rss.xml b/blog/feeds/Author-Olek-Gierczak.rss.xml new file mode 100644 index 00000000..d2251595 --- /dev/null +++ b/blog/feeds/Author-Olek-Gierczak.rss.xml @@ -0,0 +1,131 @@ + + + + PRL Blog: Posts tagged 'Author: Olek Gierczak' + PRL Blog: Posts tagged 'Author: Olek Gierczak' + http://prl.ccs.neu.edu/blog/tags/Author-Olek-Gierczak.html + Thu, 12 Dec 2019 12:51:53 UT + Thu, 12 Dec 2019 12:51:53 UT + 1800 + + PRL Offsite 2019 Retrospective + http://prl.ccs.neu.edu/blog/2019/12/12/prl-offsite-2019-retrospective/?utm_source=Author-Olek-Gierczak&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-12-12-prl-offsite-2019-retrospective + Thu, 12 Dec 2019 12:51:53 UT + Ben Greenman, Olek Gierczak + +<p>On November 11th 2019, the PRL had a private offsite meeting at the <a href="https://mtwyouth.org">More Than Words bookstore</a> in downtown Boston. For future offsite organizers, this post records what happened and how.</p> +<!-- more--> + +<h2 id="early-planning-goals">Early Planning, Goals</h2> + +<p>Every Fall, the PRL holds a kickoff meeting to assign <a href="http://prl.ccs.neu.edu/contact">roles</a> for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.</p> + +<blockquote> + <p>Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the <a href="http://www.warrencenter.com">Warren Conference Center</a> is <a href="https://www.northeastern.edu/events/northeastern-owned-off-campus-venues">Northeastern University&rsquo;s only franchised</a> off-campus venue.</p></blockquote> + +<p>The <strong>motivation</strong> for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students &mdash; both to get to know the group, and to become known. They&rsquo;re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">Spring 2017 edition</a> of HOPL.</p> + +<p>Our primary <strong>goal</strong> this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.</p> + +<p>In light of the main goal, we prefer to call the event an &ldquo;offsite&rdquo; (or &ldquo;offsite meeting&rdquo;) rather than a &ldquo;retreat&rdquo; because the target was an informative day rather than a relaxing one.</p> + +<h2 id="booking-and-logistics">Booking and Logistics</h2> + +<p>Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that&rsquo;s all we had.</p> + +<h3 id="date--nov-11">Date = Nov 11</h3> + +<p>In the beginning, we chose Veterans&rsquo; Day (2019&ndash;11&ndash;11) and Northeastern <a href="https://registrar.northeastern.edu/app/uploads/2019-2020-University-Wide-Calendar-List.pdf">reading day</a> (2019&ndash;12&ndash;05) as possible dates. We ended up with Veterans&rsquo; Day.</p> + +<p>A small number of lab members were opposed to Veterans&rsquo; Day. They gave two reasons: the Fall semester is especially busy, and Veterans&rsquo; Day is a federal holiday.</p> + +<h3 id="venue--mtw">Venue = MTW</h3> + +<p>The first venue we tried was the <a href="http://www.warrencenter.com">Warren Conference Center</a> in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the <a href="https://www.khoury.northeastern.edu/people/chelsea-smith/">Khoury Events Team</a> to reach out on our behalf. They returned with an email address and <a href="/blog/static/warren-center-meetings-and-retreats.pdf">event menu</a> (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.</p> +<!-- Donald Pepple (Northeastern) made contact with Christine Barisano (Framingham)--> + +<p>Second, we contacted the <a href="https://alumni.northeastern.edu/about/alumni-center/">Northeastern University Alumni Center</a>. They were not open on Veterans&rsquo; Day 2019.</p> + +<p>Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: <a href="https://mtwyouth.org">More Than Words</a>.</p> + +<p>More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!</p> + +<blockquote> + <p>More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you&rsquo;ll see a mix of &ldquo;employees&rdquo; and &ldquo;volunteers&rdquo; helping out.</p></blockquote> + +<p>Booking was complicated, though, by the fact that Northeastern requires <a href="http://catalog.northeastern.edu/graduate/health-sciences/academic-policies-procedures/liability-insurance/">liability insurance</a> for all off-campus events. If <strong>you</strong> are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.</p> + +<p>Here are more facts about MTW:</p> + +<ul> + <li>the theater space seats 30 with lots of extra space</li> + <li>the two long tables in the reading room can seat about 20</li> + <li>the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor</li> + <li>there&rsquo;s no whiteboard, but MTW can supply an easel stand</li> + <li>much of the furniture in store is antique and for sale, so be careful using it &mdash; both to avoid damaging it, and because antiques can be oddly-shaped</li> + <li>the bookstore was closed on Veterans&rsquo; Day, but we were able to have our event and buy books</li> + <li>MTW may have fridge space to temporarily hold leftovers</li> + <li>closest T station: <a href="https://www.mbta.com/stops/place-tumnl">Tufts Medical Center</a></li></ul> + +<h3 id="keynote--none">Keynote = None</h3> + +<p>The original, ambitious plan was to invite two keynote speakers &mdash; one working in industry and one from academia &mdash; to enrich the offsite with new knowledge. And because this was the PRL&rsquo;s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)</p> + +<p>We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.</p> + +<p>Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).</p> +<!-- to future planners: we invited Kathi Fisler; she had grant-writing plans for that day and would be interested in coming to a future offsite--> + +<h3 id="food-coffee-tea">Food, Coffee, Tea</h3> + +<p><a href="https://flourbakery.com/">Flour Bakery + Cafe</a> provided lunch. For most days, you can order from Flour using an <a href="https://flourbakery.com/menu/catering/catering-information/">online form</a>. For holidays, you may need to send an email &mdash; that&rsquo;s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan &amp; gluten-free <em>everything spiced</em> salads.</p> + +<p><a href="https://www.trycuppacoffee.com/location/">Cuppacoffee</a> provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything &mdash; the order + napkins, milk, cream cheese, butter, and knives &mdash; ourselves because Cuppacoffee is very close to MTW.</p> + +<p><a href="http://www.cmartboston.com/">CMart</a> sold us water and juices. (There is also a Whole Foods near MTW.)</p> + +<p>At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour&rsquo;s <a href="https://youtu.be/kIbhckqanHI">sticky buns</a>.</p> + +<h3 id="event-program">Event Program</h3> + +<p>The following assumptions/goals constrained the schedule for the day:</p> + +<ul> + <li>give all 17 on-campus Ph.D. students a talk slot; talks must <strong>either</strong> communicate one technical idea from recent work <strong>or</strong> say what led the speaker to apply to a PL Ph.D. program</li> + <li>allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)</li> + <li>do not make the audience sit for more than 1 hour at a time</li> + <li>maximize the number and length of breaks (to encourage discussions)</li> + <li>include a relevant and fun welcome activity</li> + <li>save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions</li></ul> + +<p>We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.</p> + +<p>The activity was <em>codewalks</em>. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition/blob/master/src/main/java/com/seriouscompany/business/java/fizzbuzz/packagenamingpackage/impl/math/arithmetics/IntegerDivider.java">IntegerDivider</a> class from a silly FizzBuzz implementation and a solution to the <a href="https://en.wikipedia.org/wiki/Dutch_national_flag_problem">Dutch national flag problem</a>). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and <a href="https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/">egoless programming</a>.</p> + +<p>The talks ran conference-style. One organizer played &ldquo;session chair&rdquo; to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.</p> + +<p>For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.</p> + +<h2 id="looking-back-at-the-details">Looking Back at the Details</h2> + +<p>Reading Day in the Spring may be a good, free date for future retreats.</p> + +<p>We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.</p> + +<p>The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we&rsquo;d keep the morning breaks short &mdash; just enough to use the restroom and grab a coffee &mdash; and aim for 30-min breaks in the afternoon.</p> + +<p>Groups of three 15-minute talks worked well, but groups of four talks might be equally good.</p> + +<p>Perhaps each speaker should get the chance to pick a talk length. <a href="https://nepls.org/">NEPLS</a>, for example, allows a choice between 5-min. and 30-min. talks.</p> + +<p>The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick &ldquo;topics&rdquo; beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.</p> +<!-- to future organizers: the panel began asking faculty for a mistake in their career and wound up with a defensive flavor--> + +<h2 id="looking-back-overall">Looking Back, Overall</h2> + +<p>This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.</p> + +<p>There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time &mdash; keep the same date &amp; program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.</p> + +<p>With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.</p> \ No newline at end of file diff --git "a/blog/feeds/Author-Olivier-Flu\314\210ckiger.atom.xml" "b/blog/feeds/Author-Olivier-Flu\314\210ckiger.atom.xml" new file mode 100644 index 00000000..883d2d2a --- /dev/null +++ "b/blog/feeds/Author-Olivier-Flu\314\210ckiger.atom.xml" @@ -0,0 +1,16 @@ + + + PRL Blog: Posts tagged 'Author: Olivier Flückiger' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Olivier-Flu-CC-88ckiger-html + 2022-02-16T00:00:42Z + + [Ř Overview I (cross-post)](https://www.o1o.ch/lab/entry/1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc.html) + + urn:http-prl-ccs-neu-edu:-blog-2022-02-16-r-CC-8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html + 2022-02-16T00:00:42Z + 2022-02-16T00:00:42Z + + Olivier Flückiger + \ No newline at end of file diff --git "a/blog/feeds/Author-Olivier-Flu\314\210ckiger.rss.xml" "b/blog/feeds/Author-Olivier-Flu\314\210ckiger.rss.xml" new file mode 100644 index 00000000..3ce2f7f3 --- /dev/null +++ "b/blog/feeds/Author-Olivier-Flu\314\210ckiger.rss.xml" @@ -0,0 +1,16 @@ + + + + PRL Blog: Posts tagged 'Author: Olivier Flückiger' + PRL Blog: Posts tagged 'Author: Olivier Flückiger' + http://prl.ccs.neu.edu/blog/tags/Author-Olivier-Flu%CC%88ckiger.html + Wed, 16 Feb 2022 00:00:42 UT + Wed, 16 Feb 2022 00:00:42 UT + 1800 + + [Ř Overview I (cross-post)](https://www.o1o.ch/lab/entry/1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc.html) + http://prl.ccs.neu.edu/blog/2022/02/16/-r%CC%8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/?utm_source=Author-Olivier-Flu%CC%88ckiger&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-02-16-r-CC-8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html + Wed, 16 Feb 2022 00:00:42 UT + Olivier Flückiger + \ No newline at end of file diff --git a/blog/feeds/Author-Rob-Kleffner.atom.xml b/blog/feeds/Author-Rob-Kleffner.atom.xml new file mode 100644 index 00000000..b4ed7beb --- /dev/null +++ b/blog/feeds/Author-Rob-Kleffner.atom.xml @@ -0,0 +1,34 @@ + + + PRL Blog: Posts tagged 'Author: Rob Kleffner' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Rob-Kleffner-html + 2017-03-10T16:23:30Z + + Type Inference in Stack-Based Programming Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-03-10-type-inference-in-stack-based-programming-languages + 2017-03-10T16:23:30Z + 2017-03-10T16:23:30Z + + Rob Kleffner + <!-- more--> + +<p>Stack-based languages occupy a niche in today&rsquo;s programming language environment. The predominant stack-based language in use by programmers is Forth, and is found mostly on embedded devices. These languages also find use as compile targets for more popular languages: the CIL and JVM are both stack-based. Less popular but highly interesting languages to mention include <a href="http://www.kevinalbrecht.com/code/joy-mirror/joy.html">Joy</a> and <a href="http://factorcode.org/">Factor</a>, known for their emphasis on higher-order stack-based programming.</p> + +<p>The majority of stack-based languages are not statically typed, and it would be a stretch to call Forth even dynamically typed. As such, developing large projects in Forth or Factor can require great discipline on the part of the programmer to avoid type errors.</p> + +<p>In this talk, I presented the development of type inference for stack-based languages as a linear sequence, divided into two overarching segments:</p> + +<ul> + <li>An algebraic system known as <em>stack effects</em></li> + <li>Systems that can be encoded as <em>nested pairs</em> in standard functional programming languages</li></ul> + +<p>The thread of research on stack effects began with Jaanus Pöial in the early 1990&rsquo;s, and is a formalization of a commenting style well-known in the Forth community. The nested tuple systems were first examined by Okasaki in 1993 in the context of Haskell, and were later applied to higher-order stack-based languages. At the end, I give some avenues for extending the research on these systems, and list some pitfalls to be avoided in further research.</p> + +<p>Full notes (as PDF documents) &mdash; see the <a href="https://github.com/nuprl/hopl-s2017/tree/master/type-inference-for-stack-languages">git repository</a> for more documents:</p> + +<ul> + <li><a href="/blog/static/stack-languages-talk-notes.pdf">Talk notes</a></li> + <li><a href="/blog/static/stack-languages-annotated-bib.pdf">Annotated bibliography</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Rob-Kleffner.rss.xml b/blog/feeds/Author-Rob-Kleffner.rss.xml new file mode 100644 index 00000000..aff2696a --- /dev/null +++ b/blog/feeds/Author-Rob-Kleffner.rss.xml @@ -0,0 +1,34 @@ + + + + PRL Blog: Posts tagged 'Author: Rob Kleffner' + PRL Blog: Posts tagged 'Author: Rob Kleffner' + http://prl.ccs.neu.edu/blog/tags/Author-Rob-Kleffner.html + Fri, 10 Mar 2017 16:23:30 UT + Fri, 10 Mar 2017 16:23:30 UT + 1800 + + Type Inference in Stack-Based Programming Languages + http://prl.ccs.neu.edu/blog/2017/03/10/type-inference-in-stack-based-programming-languages/?utm_source=Author-Rob-Kleffner&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-03-10-type-inference-in-stack-based-programming-languages + Fri, 10 Mar 2017 16:23:30 UT + Rob Kleffner + <!-- more--> + +<p>Stack-based languages occupy a niche in today&rsquo;s programming language environment. The predominant stack-based language in use by programmers is Forth, and is found mostly on embedded devices. These languages also find use as compile targets for more popular languages: the CIL and JVM are both stack-based. Less popular but highly interesting languages to mention include <a href="http://www.kevinalbrecht.com/code/joy-mirror/joy.html">Joy</a> and <a href="http://factorcode.org/">Factor</a>, known for their emphasis on higher-order stack-based programming.</p> + +<p>The majority of stack-based languages are not statically typed, and it would be a stretch to call Forth even dynamically typed. As such, developing large projects in Forth or Factor can require great discipline on the part of the programmer to avoid type errors.</p> + +<p>In this talk, I presented the development of type inference for stack-based languages as a linear sequence, divided into two overarching segments:</p> + +<ul> + <li>An algebraic system known as <em>stack effects</em></li> + <li>Systems that can be encoded as <em>nested pairs</em> in standard functional programming languages</li></ul> + +<p>The thread of research on stack effects began with Jaanus Pöial in the early 1990&rsquo;s, and is a formalization of a commenting style well-known in the Forth community. The nested tuple systems were first examined by Okasaki in 1993 in the context of Haskell, and were later applied to higher-order stack-based languages. At the end, I give some avenues for extending the research on these systems, and list some pitfalls to be avoided in further research.</p> + +<p>Full notes (as PDF documents) &mdash; see the <a href="https://github.com/nuprl/hopl-s2017/tree/master/type-inference-for-stack-languages">git repository</a> for more documents:</p> + +<ul> + <li><a href="/blog/static/stack-languages-talk-notes.pdf">Talk notes</a></li> + <li><a href="/blog/static/stack-languages-annotated-bib.pdf">Annotated bibliography</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Author-Sam-Caldwell.atom.xml b/blog/feeds/Author-Sam-Caldwell.atom.xml new file mode 100644 index 00000000..c776f6c3 --- /dev/null +++ b/blog/feeds/Author-Sam-Caldwell.atom.xml @@ -0,0 +1,1344 @@ + + + PRL Blog: Posts tagged 'Author: Sam Caldwell' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Sam-Caldwell-html + 2018-10-22T15:05:17Z + + Defining Local Bindings in Turnstile Languages + + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + 2018-10-22T15:05:17Z + 2018-10-22T15:05:17Z + + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> + + PLT Redex FAQ + + urn:http-prl-ccs-neu-edu:-blog-2017-09-25-plt-redex-faq + 2017-09-25T23:39:16Z + 2017-09-25T23:39:16Z + Ben Greenman + Sam Caldwell + + Ben Greenman, Sam Caldwell + +<p>A short guide to Redex concepts, conventions, and common mistakes.</p> +<!-- more--> + +<hr /> + +<p><em>To contribute to this FAQ, submit issues and pull requests to: <a href="https://github.com/nuprl/website/">https://github.com/nuprl/website/</a></em></p> + +<h3 id="q-what-is-redex-useful-for">Q. What is Redex useful for?</h3> + +<ol> + <li>declaring <a href="https://en.wikipedia.org/wiki/Regular_tree_grammar">regular tree grammars</a></li> + <li>defining <em>pattern</em>-based judgments and relations on <em>terms</em></li> + <li>testing properties of the above</li></ol> + +<p>More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.</p> + +<h3 id="q-what-is-redex-not-useful-for">Q. What is Redex <strong>not</strong> useful for?</h3> + +<p>Proving theorems about a grammar, judgment, or relation.</p> + +<h3 id="q-what-is-a-term">Q. What is a <em>term</em>?</h3> + +<p>Informally, a term is:</p> + +<ul> + <li>a Redex &ldquo;atom&rdquo;, or</li> + <li>an object that represents a sequence of characters.</li></ul> + +<p>More formally, a term is the result of evaluating <strong>(term X)</strong>, where <strong>X</strong> is any syntactically-correct Racket expression.</p> + +<p>Examples:</p> + +<pre><code>$ racket +Welcome to Racket v6.10.0.3. +&gt; (require redex/reduction-semantics) +&gt; (term 42) +42 +&gt; (term (+ 2 2)) +'(+ 2 2) +&gt; (term ("hello" world (#false))) +'("hello" world (#f))</code></pre> + +<p>Some terms may look strange. That&rsquo;s OK, because a term by itself has no meaning.</p> + +<p>Terms can refer to previously-defined values using the <strong>unquote</strong> escape.</p> + +<pre><code>&gt; (define x (term 42)) +&gt; (term (+ 2 x)) +'(+ 2 x) +&gt; (term (+ ,x (unquote x))) +'(+ 42 42)</code></pre> + +<h3 id="q-what-is-a-redex-model">Q. What is a <em>Redex model</em>?</h3> + +<p>A Redex model is collection of tools for working with terms. The tools may include:</p> + +<ul> + <li><em>languages</em>, to define a grammar for terms</li> + <li><em>judgments</em>, to describe properties of terms or relations between terms</li> + <li><em>metafunctions</em>, to transform terms</li> + <li><em>reduction relations</em>, to define a term rewriting system</li></ul> + +<p>The goal of these tools is to encode a &ldquo;real thing&rdquo; (maybe, a programming language) using Redex terms.</p> + +<h3 id="q-what-is-a-language">Q. What is a language?</h3> + +<p>A Redex <em>language</em> is a named set of non-terminals, <em>patterns</em>, and <em>binding forms</em>. For example, a Redex model of the natural numbers might start with this language:</p> + +<pre><code>(define-language nat + [N ::= Zero + (Plus1 N)])</code></pre> + +<ul> + <li>the name of the language is <strong>nat</strong></li> + <li>the non-terminal <strong>N</strong> is associated with two patterns: <strong>Zero</strong> and <strong>(Plus1 N)</strong></li> + <li>there are no <em>binding forms</em></li></ul> + +<p>Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.</p> + +<p>The non-terminal <strong>N</strong> describes all terms that are either:</p> + +<ol> + <li>the symbol <strong>Zero</strong></li> + <li>lists of the form <strong>(Plus1 N)</strong>, where <strong>N</strong> is either <strong>Zero</strong> or another &ldquo;Plus1&rdquo;</li></ol> + +<p>For example,</p> + +<pre><code>(term Zero) +(term (Plus1 Zero)) +(term (Plus1 (Plus1 Zero))) +(term (Plus1 (Plus1 (Plus1 Zero)))) +;; .... and so on</code></pre> + +<p>If a language has binding forms, then some terms can introduce names. See the question on <em>binding forms</em> (below) for an example.</p> + +<h3 id="q-what-is-a-pattern">Q. What is a pattern?</h3> + +<p>A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains <em>named non-terminals</em> from the language, then you can ask whether a Redex term matches the pattern.</p> + +<p>A <em>named non-terminal</em> for a language <strong>L</strong> is an identifier made of: (1) a non-terminal from <strong>L</strong>, (2) an underscore (<strong>_</strong>), and (3) any other identifier. See the FAQ entry below.</p> + +<p>For example, <strong>(redex-match? L p t)</strong> returns <strong>#true</strong> if the term <strong>t</strong> matches the pattern <strong>p</strong> relative to the language <strong>L</strong>.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(redex-match? nat N_some-name (term Zero)) +;; #true +(redex-match? nat (Plus1 N_a) (term Zero)) +;; #false +(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; #true</code></pre> + +<p>If <strong>(redex-match? L p t)</strong> is <strong>#true</strong>, then <strong>(redex-match L p t)</strong> shows how named non-terminals in the pattern bind to subterms of <strong>t</strong>.</p> + +<pre><code>(redex-match nat N_0 (term Zero)) +;; (list (match (list (bind 'N_0 'Zero)))) +(redex-match nat (Plus1 N_0) (term Zero)) +;; #f +(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))</code></pre> + +<h3 id="q-what-is-a-named-non-terminal">Q. What is a named non-terminal?</h3> + +<p>A named non-terminal in a language <strong>L</strong> is an identifier of the form <strong>X_Y</strong>, where:</p> + +<ul> + <li><strong>X</strong> is a non-terminal from <strong>L</strong></li> + <li><strong>Y</strong> is any identifier</li></ul> + +<p>The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.</p> + +<pre><code>(define-language trees + [binary-tree ::= Leaf + (Node binary-tree binary-tree)]) + +(redex-match trees + (Node binary-tree_left binary-tree_right) + (term (Node Leaf (Node Leaf Leaf)))) +;; (list +;; (match +;; (list (bind 'binary-tree_left 'Leaf) +;; (bind 'binary-tree_right '(Node Leaf Leaf)))))</code></pre> + +<h3 id="q-what-else-can-patterns-express">Q. What else can patterns express?</h3> + +<p>Redex patterns may contain special identifiers to guide pattern-matching. For instance:</p> + +<ul> + <li>The <strong>_</strong> pattern matches any term (and does not bind).</li> + <li>A pattern <strong>(p &hellip;)</strong> matches any sequence whose elements match the pattern <strong>p</strong>. If the pattern <strong>p</strong> is a named non-terminal, then the non-terminal binds a sequence of subterms.</li></ul> + +<p>Examples:</p> + +<pre><code>(redex-match? nat (Plus1 _) (term (Plus1 9))) +;; #true +(redex-match? nat (N_0 ...) (term ())) +;; #true +(redex-match? nat (N_0 ...) (term (Zero))) +;; #true +(redex-match nat (N_0 ...) (term (Zero Zero Zero))) +;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))</code></pre> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29">the Redex reference</a> for the full pattern language, including the <em>named and unique non-terminals</em> of the form <strong>X_!_Y</strong>.</p> + +<h3 id="q-what-can-patterns-not-express">Q. What can patterns <strong>not</strong> express?</h3> + +<ul> + <li>Disjunctions of patterns, e.g., &ldquo;number or boolean&rdquo;. (Use a language non-terminal.)</li> + <li>Negations of patterns. (Compose <strong>not</strong> with <strong>redex-match?</strong>.)</li> + <li>Some non-regular patterns. (Named dots <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29"><code>..._N</code></a> or <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-language%29%29"><code>define-language</code></a> with a side condition may be able to help.)</li></ul> + +<h3 id="q-what-is-a-judgment">Q. What is a judgment?</h3> + +<p>A Redex <em>judgment</em> form defines a relation on terms. The relation is defined by a set of inference rules.</p> + +<p>Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:</p> + +<ul> + <li>well-formedness conditions (i.e., whether a term contains free variables)</li> + <li>type checking rules</li> + <li>type inference rules</li> + <li>evaluation relations</li></ul> + +<p>Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(define-judgment-form nat + #:mode (N= I I) + #:contract (N= N N) + [ + --- Zero= + (N= Zero Zero)] + [ + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1) + (N= N_0-- N_1--) + --- Plus1= + (N= N_0 N_1)])</code></pre> + +<ol> + <li>the language is <strong>nat</strong>; Redex uses the language to interpret patterns</li> + <li>the mode is <strong>(N= I I)</strong>; this means <strong>N=</strong> is the name of a judgment that expects two input terms (or, <strong>N=</strong> is a binary relation on terms)</li> + <li>the contract is <strong>(N= N N)</strong>; in other words, <strong>N=</strong> expects two terms that match the <strong>N</strong> non-terminal from the <strong>nat</strong> language</li> + <li>there are two inference rules, named <strong>Zero=</strong> and <strong>Plus1=</strong></li> + <li>the <strong>Zero=</strong> rule states that <strong>(N= Zero Zero)</strong> always holds</li> + <li>the <strong>Plus1=</strong> rule states that <strong>(N= N_0 N_1)</strong> holds if <strong>N_0</strong> and <strong>N_1</strong> are both <strong>Plus1</strong> terms whose contents are related by <strong>N=</strong></li></ol> + +<p>The <strong>where</strong> clauses are <em>guards</em>. When Redex tries to apply a rule with premises of the form <strong>(where pattern term)</strong>, it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-judgment-form%29%29">the Redex reference</a> for more.</p> + +<pre><code>(judgment-holds (N= Zero Zero)) +;; #true +(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Note: the inference rules form a <em>set</em>, not a <em>sequence</em>. So when you ask Redex whether <strong>(judgment-holds (N= Zero Zero))</strong>, it applies all rules that match <strong>(N= Zero Zero)</strong>. For <strong>N=</strong> this is just one rule, but in general it could be many rules.</p> + +<h3 id="q-what-is-a-judgment-form-mode">Q. What is a judgment form <strong>#:mode</strong>?</h3> + +<p>A <strong>#:mode</strong> declaration expects a list of the form <strong>(id pos-use &hellip;)</strong>, where <strong>id</strong> is an identifier and each <strong>pos-use</strong> is either <strong>I</strong> or <strong>O</strong>. These declarations say four things:</p> + +<ol> + <li><strong>id</strong> is the name of a new judgment form</li> + <li><strong>id</strong> expects <strong>N</strong> arguments, where <strong>N</strong> is the number of <strong>pos-use</strong> symbols</li> + <li><strong>id</strong> expects an <em>input</em> at each position where the mode contains an <strong>I</strong></li> + <li><strong>id</strong> produces an <em>output</em> at each position where the mode contains an <strong>O</strong></li></ol> + +<p>For example, a type inference judgment may take an expression as input and output a type. Here&rsquo;s a fast and easy type inference judgment for arithmetic expressions. Given any term <strong>e_0</strong>, the judgment outputs the type <strong>Int</strong>.</p> + +<pre><code>(define-language Arith + (e ::= integer (e + e)) + (τ ::= Int)) + +(define-judgment-form Arith + #:mode (infer-type I O) + #:contract (infer-type e τ) + [ + --- T-Int + (infer-type e_0 Int)])</code></pre> + +<h3 id="q-what-can-judgments-not-express">Q. What can judgments <strong>not</strong> express?</h3> + +<p>Redex does not support inference rules that require guessing.</p> + +<p>One example of this is a transitivity rule: "<strong>τ_0</strong> is related to <strong>τ_2</strong> if there exists a <strong>τ_1</strong> such that <strong>τ_0</strong> is related to <strong>τ_1</strong> and <strong>τ_1</strong> is related to <strong>τ_2</strong>". The following example tries to define a transitive subtyping (<strong>&lt;:</strong>) relation, but Redex rejects the <strong>S-Trans</strong> rule.</p> + +<pre><code>(define-language SomeTypes + (τ ::= (→ τ τ) Integer)) + +(define-judgment-form SomeTypes + #:mode (&lt;: I I) + #:contract (&lt;: τ τ) + [ + (&lt;: τ_0 τ_1) + (&lt;: τ_1 τ_2) + --- S-Trans + (&lt;: τ_0 τ_2)] + [ + --- S-Refl + (&lt;: τ_0 τ_0)] + [ + (&lt;: τ_dom-1 τ_dom-0) + (&lt;: τ_cod-0 τ_cod-1) + --- S-Arrow + (&lt;: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])</code></pre> + +<p>The error is that in the rule <strong>S-Trans</strong>, the named non-terminal <strong>τ_1</strong> appears in an input position but is not bound to a term.</p> + +<h3 id="q-what-is-a-metafunction">Q. What is a metafunction?</h3> + +<p>A metafunction is a term-level function on terms.</p> + +<p>Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.</p> + +<p>Here is a metafunction that returns <strong>#true</strong> when given two equal natural numbers. The definition is similar to the <strong>N=</strong> judgment form.</p> + +<pre><code>(define-metafunction nat + N=? : N N -&gt; boolean + [(N=? Zero Zero) + #true] + [(N=? N_0 N_1) + (N=? N_0-- N_1--) + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1)] + [(N=? N_0 N_1) + #false])</code></pre> + +<ul> + <li>the metafunction is named <strong>N=?</strong></li> + <li>its contract is <strong>N N -&gt; boolean</strong>, this means <strong>N=?</strong> expects 2 terms that match the <strong>N</strong> pattern and returns a term that matches the pattern <strong>boolean</strong></li> + <li>there are three cases; the second case is guarded by two <strong>where</strong> clauses</li></ul> + +<p>Any occurrence of <strong>(N=? &hellip;.)</strong> in any term is evaluated.</p> + +<pre><code>(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(term ((N=? Zero Zero) Zero)) +;; '(#true Zero) +(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Any occurrence of <strong>N=?</strong> outside a <strong>term</strong> is an error.</p> + +<p>Metafunction <strong>where</strong>-clauses are analogous to judgment form <strong>where</strong>-clauses. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._metafunction%29">the Redex reference</a> for more.</p> + +<p>Note: the cases in a metafunction form a <em>sequence</em>, not a <em>set</em>. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.</p> + +<h3 id="q-should-i-use-a-metafunction-or-a-judgment-form">Q. Should I use a metafunction or a judgment form?</h3> + +<p>Use a judgment form.</p> + +<p>Metafunctions are handy, but judgments are easier to read and debug and maintain.</p> + +<h3 id="q-what-is-a-reduction-relation">Q. What is a reduction relation?</h3> + +<p>A reduction relation is a set of term-rewriting rules.</p> + +<p>Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.</p> + +<ul> + <li>The language tells Redex how to interpret patterns.</li> + <li>The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.</li> + <li>The rewrite rules have the form <strong>(&mdash;&gt; term term guard &hellip;)</strong>. The term on the left is the input, the term on the right is the output.</li></ul> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._reduction-relation%29%29">the Redex reference</a> for a full description of the guards.</p> + +<p>The preferred way to define a reduction relation is to define a language that includes three non-terminals:</p> + +<ol> + <li>a non-terminal for the domain of the reduction relation</li> + <li>a non-terminal for a <em>subset</em> of the domain that cannot reduce further</li> + <li>a non-terminal for evaluation contexts</li></ol> + +<p>An evaluation context is a term that contains a <strong>hole</strong>. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite &mdash; in particular, the sub-term that matches the <strong>hole</strong>.</p> + +<p>In the following example, <strong>bexp</strong> is the domain of a reduction relation. A <strong>bexp</strong> term represents a boolean expression, which can be <strong>#true</strong> or <strong>#false</strong> or a conjunction of expressions or a disjunction of expressions. The boolean expressions <strong>#true</strong> and <strong>#false</strong> are also values (<strong>val</strong>); these cannot reduce further. The non-terminal <strong>E</strong> is for evaluation contexts.</p> + +<pre><code>(define-language Bool + (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp)) + (val ::= #true #false) + (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E))) + +(define step + (reduction-relation Bool + #:domain bexp + [--&gt; (in-hole E (val_lhs ∧ val_rhs)) + (in-hole E val_new) + ∧-step + (where val_new ,(and (term val_lhs) (term val_rhs)))] + [--&gt; (in-hole E (val_lhs ∨ val_rhs)) + (in-hole E val_new) + ∨-step + (where val_new ,(or (term val_lhs) (term val_rhs)))]))</code></pre> + +<p>The function <strong>apply-reduction-relation</strong> applies a reduction relation to a term and returns a list of ways that the term can step.</p> + +<pre><code>(apply-reduction-relation step (term #true)) +;; '() +(apply-reduction-relation step (term (#true ∧ #true))) +;; '(#true) +(apply-reduction-relation step (term (#true ∧ #false))) +;; '(#false) +(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true))) +;; '((#true ∧ #true))</code></pre> + +<p>Three things about the reduction relation <strong>step</strong>:</p> + +<ol> + <li>Using <strong>in-hole</strong> on the first argument of <strong>&mdash;&gt;</strong> searches a term for a subterm that Redex can apply a reduction rule to.</li> + <li>Using <strong>in-hole</strong> on the second argument of <strong>&mdash;&gt;</strong> puts a new value back into the <strong>hole</strong> in the evaluation context.</li> + <li>The unquote operator (<strong>,</strong>) escapes to &ldquo;Racket mode&rdquo; (see below) to evaluate a conjunction or disjunction.</li></ol> + +<p>A judgment or metafunction is a formal alternative to &ldquo;escaping to Racket&rdquo;, but escaping can be convenient.</p> + +<p>Note: the cases in a reduction relation form a <em>set</em>, not a <em>sequence</em>. If more than one case matches, Redex applies them all.</p> + +<h3 id="q-what-is-racket-mode-what-is-redex-mode">Q. What is &ldquo;Racket mode&rdquo;? What is &ldquo;Redex mode&rdquo;?</h3> + +<p>Code in a Redex model is sometimes evaluated in &ldquo;Racket mode&rdquo; and sometimes evaluated in &ldquo;Redex mode&rdquo;. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.</p> + +<p>Key points:</p> + +<ol> + <li>A Redex program starts in Racket mode.</li> + <li>The <strong>X</strong> in <strong>(term X)</strong> is evaluated in Redex mode &hellip;</li> + <li>&hellip; unless <strong>X</strong> contains unquoted sub-expressions. Unquoting escapes to Racket mode &hellip;</li> + <li>&hellip; and <strong>term</strong>s inside an unquoted sub-expression are evaluated in Redex mode.</li></ol> + +<p>In other words, <strong>term</strong> enters Redex mode and <strong>unquote</strong> (<strong>,</strong>) escapes back to Racket.</p> + +<p>Redex implicitly switches to Redex mode in a few other places, for example:</p> + +<ul> + <li>the right-side of a <strong>where</strong> clause is in Redex mode</li> + <li>the result of a metafunction is in Redex mode</li></ul> + +<p>When in doubt, try using an <strong>unquote</strong>. Redex will raise an exception if it finds an unquote in Racket mode.</p> + +<h3 id="q-are-side-conditions-evaluated-in-racket-mode-or-redex-mode">Q. Are <strong>side-condition</strong>s evaluated in &ldquo;Racket mode&rdquo; or &ldquo;Redex mode&rdquo;?</h3> + +<p>A <strong>(side-condition e)</strong> sometimes evaluates <strong>e</strong> as a Racket expression and sometimes evaluates <strong>e</strong> as a Redex expression.</p> + +<ul> + <li>reduction relations and metafunctions expect a <strong>Racket</strong> expression</li> + <li>judgments expect a <strong>Redex</strong> expression</li></ul> + +<h3 id="q-what-is-a-binding-form">Q. What is a binding form?</h3> + +<p>In the lambda calculus, <strong>λ</strong>-terms bind variables. A term <strong>(λ x M)</strong> means that any free occurrence of <strong>x</strong> in the sub-term <strong>M</strong> refers to the <strong>x</strong> from the <strong>λ</strong>-term.</p> + +<p>Redex can express this idea with a binding form.</p> + +<pre><code>(define-language Λ + [e ::= (e e) x (λ x e)] + [x ::= variable-not-otherwise-mentioned] + #:binding-forms + (λ x_0 e_0 #:refers-to x_0))</code></pre> + +<p>Note: all the non-terminals in a language must be defined before the <strong>#:binding-forms</strong> keyword. If a non-terminal definition appears after the <strong>#:binding-forms</strong> keyword, then Redex will interpret the &ldquo;definition&rdquo; as a binding form.</p> + +<p>Binding forms work together with Redex&rsquo;s functions for substitution and alphabetic equivalence.</p> + +<pre><code>(alpha-equivalent? Λ + (term (λ x x)) + (term (λ y y)))) +;; #true + +(define-metafunction Λ + test-substitute : e -&gt; e + [(test-substitute (λ x_0 e_0)) + (substitute e_0 x_0 y)]) +(term (test-substitute (λ z (z z)))) +;; '(y y)</code></pre> + +<h3 id="q-what-is--what-is-">Q. What is <strong>&hellip;</strong>? What is <strong>&hellip;.</strong>?</h3> + +<p>Three dots (<strong>&hellip;</strong>) is for building patterns. If <strong>p</strong> is a pattern then <strong>(p &hellip;)</strong> matches any list whose elements all match <strong>p</strong>.</p> + +<pre><code>(define-language L) +(redex-match? L (number ... boolean ...) (term (1 2 #true #true))) +;; #true</code></pre> + +<p>Four dots (<strong>&hellip;.</strong>) may be used in <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-extended-language%29%29"><strong>define-extended-language</strong></a> to extend a previously-defined non-terminal.</p> + +<pre><code>(define-language C + (keyword ::= auto break case)) +(define-extended-language C++ + C + (keyword ::= .... class)) + +(redex-match? C keyword (term auto)) +;; #true +(redex-match? C keyword (term class)) +;; #false +(redex-match? C++ keyword (term auto)) +;; #true +(redex-match? C++ keyword (term class)) +;; #true</code></pre> + +<h3 id="q-where-to-learn-more-about-redex">Q. Where to learn more about Redex?</h3> + +<p>&ldquo;Critical path&rdquo; resources:</p> + +<ul> + <li>Code examples for this post: <a href="https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt">https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt</a></li> + <li>Redex documentation: <a href="http://docs.racket-lang.org/redex/index.html">http://docs.racket-lang.org/redex/index.html</a></li> + <li><a href="https://dvanhorn.github.io/redex-aam-tutorial/"><em>An Introduction to Redex with Abstracting Abstract Machines</em></a> by David Van Horn</li> + <li><a href="https://www.leafac.com/playing-the-game-with-plt-redex/"><em>Playing the Game with PLT Redex</em></a> by Leandro Facchinetti</li> + <li><a href="https://williamjbowman.com/doc/experimenting-with-redex/index.html"><em>Experimenting with Languages in Redex</em></a> by William J. Bowman</li></ul> + +<p>&ldquo;Procrastination&rdquo; resources:</p> + +<ul> + <li><a href="http://tata.gforge.inria.fr/"><em>Tree Automata Techniques and Applications</em></a></li> + <li><a href="http://lamport.azurewebsites.net/pubs/lamport-types.pdf"><em>Should your Specification Language be Typed?</em></a></li> + <li>Redex source code (see <code>redex-lib/</code>): <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Fall 2016 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + 2017-01-02T16:39:37Z + 2017-01-02T16:39:37Z + Ben Chung + Milo Davis + Ming-Ho Yee + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Sam-Caldwell.rss.xml b/blog/feeds/Author-Sam-Caldwell.rss.xml new file mode 100644 index 00000000..d972d243 --- /dev/null +++ b/blog/feeds/Author-Sam-Caldwell.rss.xml @@ -0,0 +1,1324 @@ + + + + PRL Blog: Posts tagged 'Author: Sam Caldwell' + PRL Blog: Posts tagged 'Author: Sam Caldwell' + http://prl.ccs.neu.edu/blog/tags/Author-Sam-Caldwell.html + Mon, 22 Oct 2018 15:05:17 UT + Mon, 22 Oct 2018 15:05:17 UT + 1800 + + Defining Local Bindings in Turnstile Languages + http://prl.ccs.neu.edu/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/?utm_source=Author-Sam-Caldwell&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + Mon, 22 Oct 2018 15:05:17 UT + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> + + PLT Redex FAQ + http://prl.ccs.neu.edu/blog/2017/09/25/plt-redex-faq/?utm_source=Author-Sam-Caldwell&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-25-plt-redex-faq + Mon, 25 Sep 2017 23:39:16 UT + Ben Greenman, Sam Caldwell + +<p>A short guide to Redex concepts, conventions, and common mistakes.</p> +<!-- more--> + +<hr /> + +<p><em>To contribute to this FAQ, submit issues and pull requests to: <a href="https://github.com/nuprl/website/">https://github.com/nuprl/website/</a></em></p> + +<h3 id="q-what-is-redex-useful-for">Q. What is Redex useful for?</h3> + +<ol> + <li>declaring <a href="https://en.wikipedia.org/wiki/Regular_tree_grammar">regular tree grammars</a></li> + <li>defining <em>pattern</em>-based judgments and relations on <em>terms</em></li> + <li>testing properties of the above</li></ol> + +<p>More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.</p> + +<h3 id="q-what-is-redex-not-useful-for">Q. What is Redex <strong>not</strong> useful for?</h3> + +<p>Proving theorems about a grammar, judgment, or relation.</p> + +<h3 id="q-what-is-a-term">Q. What is a <em>term</em>?</h3> + +<p>Informally, a term is:</p> + +<ul> + <li>a Redex &ldquo;atom&rdquo;, or</li> + <li>an object that represents a sequence of characters.</li></ul> + +<p>More formally, a term is the result of evaluating <strong>(term X)</strong>, where <strong>X</strong> is any syntactically-correct Racket expression.</p> + +<p>Examples:</p> + +<pre><code>$ racket +Welcome to Racket v6.10.0.3. +&gt; (require redex/reduction-semantics) +&gt; (term 42) +42 +&gt; (term (+ 2 2)) +'(+ 2 2) +&gt; (term ("hello" world (#false))) +'("hello" world (#f))</code></pre> + +<p>Some terms may look strange. That&rsquo;s OK, because a term by itself has no meaning.</p> + +<p>Terms can refer to previously-defined values using the <strong>unquote</strong> escape.</p> + +<pre><code>&gt; (define x (term 42)) +&gt; (term (+ 2 x)) +'(+ 2 x) +&gt; (term (+ ,x (unquote x))) +'(+ 42 42)</code></pre> + +<h3 id="q-what-is-a-redex-model">Q. What is a <em>Redex model</em>?</h3> + +<p>A Redex model is collection of tools for working with terms. The tools may include:</p> + +<ul> + <li><em>languages</em>, to define a grammar for terms</li> + <li><em>judgments</em>, to describe properties of terms or relations between terms</li> + <li><em>metafunctions</em>, to transform terms</li> + <li><em>reduction relations</em>, to define a term rewriting system</li></ul> + +<p>The goal of these tools is to encode a &ldquo;real thing&rdquo; (maybe, a programming language) using Redex terms.</p> + +<h3 id="q-what-is-a-language">Q. What is a language?</h3> + +<p>A Redex <em>language</em> is a named set of non-terminals, <em>patterns</em>, and <em>binding forms</em>. For example, a Redex model of the natural numbers might start with this language:</p> + +<pre><code>(define-language nat + [N ::= Zero + (Plus1 N)])</code></pre> + +<ul> + <li>the name of the language is <strong>nat</strong></li> + <li>the non-terminal <strong>N</strong> is associated with two patterns: <strong>Zero</strong> and <strong>(Plus1 N)</strong></li> + <li>there are no <em>binding forms</em></li></ul> + +<p>Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.</p> + +<p>The non-terminal <strong>N</strong> describes all terms that are either:</p> + +<ol> + <li>the symbol <strong>Zero</strong></li> + <li>lists of the form <strong>(Plus1 N)</strong>, where <strong>N</strong> is either <strong>Zero</strong> or another &ldquo;Plus1&rdquo;</li></ol> + +<p>For example,</p> + +<pre><code>(term Zero) +(term (Plus1 Zero)) +(term (Plus1 (Plus1 Zero))) +(term (Plus1 (Plus1 (Plus1 Zero)))) +;; .... and so on</code></pre> + +<p>If a language has binding forms, then some terms can introduce names. See the question on <em>binding forms</em> (below) for an example.</p> + +<h3 id="q-what-is-a-pattern">Q. What is a pattern?</h3> + +<p>A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains <em>named non-terminals</em> from the language, then you can ask whether a Redex term matches the pattern.</p> + +<p>A <em>named non-terminal</em> for a language <strong>L</strong> is an identifier made of: (1) a non-terminal from <strong>L</strong>, (2) an underscore (<strong>_</strong>), and (3) any other identifier. See the FAQ entry below.</p> + +<p>For example, <strong>(redex-match? L p t)</strong> returns <strong>#true</strong> if the term <strong>t</strong> matches the pattern <strong>p</strong> relative to the language <strong>L</strong>.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(redex-match? nat N_some-name (term Zero)) +;; #true +(redex-match? nat (Plus1 N_a) (term Zero)) +;; #false +(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; #true</code></pre> + +<p>If <strong>(redex-match? L p t)</strong> is <strong>#true</strong>, then <strong>(redex-match L p t)</strong> shows how named non-terminals in the pattern bind to subterms of <strong>t</strong>.</p> + +<pre><code>(redex-match nat N_0 (term Zero)) +;; (list (match (list (bind 'N_0 'Zero)))) +(redex-match nat (Plus1 N_0) (term Zero)) +;; #f +(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))</code></pre> + +<h3 id="q-what-is-a-named-non-terminal">Q. What is a named non-terminal?</h3> + +<p>A named non-terminal in a language <strong>L</strong> is an identifier of the form <strong>X_Y</strong>, where:</p> + +<ul> + <li><strong>X</strong> is a non-terminal from <strong>L</strong></li> + <li><strong>Y</strong> is any identifier</li></ul> + +<p>The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.</p> + +<pre><code>(define-language trees + [binary-tree ::= Leaf + (Node binary-tree binary-tree)]) + +(redex-match trees + (Node binary-tree_left binary-tree_right) + (term (Node Leaf (Node Leaf Leaf)))) +;; (list +;; (match +;; (list (bind 'binary-tree_left 'Leaf) +;; (bind 'binary-tree_right '(Node Leaf Leaf)))))</code></pre> + +<h3 id="q-what-else-can-patterns-express">Q. What else can patterns express?</h3> + +<p>Redex patterns may contain special identifiers to guide pattern-matching. For instance:</p> + +<ul> + <li>The <strong>_</strong> pattern matches any term (and does not bind).</li> + <li>A pattern <strong>(p &hellip;)</strong> matches any sequence whose elements match the pattern <strong>p</strong>. If the pattern <strong>p</strong> is a named non-terminal, then the non-terminal binds a sequence of subterms.</li></ul> + +<p>Examples:</p> + +<pre><code>(redex-match? nat (Plus1 _) (term (Plus1 9))) +;; #true +(redex-match? nat (N_0 ...) (term ())) +;; #true +(redex-match? nat (N_0 ...) (term (Zero))) +;; #true +(redex-match nat (N_0 ...) (term (Zero Zero Zero))) +;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))</code></pre> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29">the Redex reference</a> for the full pattern language, including the <em>named and unique non-terminals</em> of the form <strong>X_!_Y</strong>.</p> + +<h3 id="q-what-can-patterns-not-express">Q. What can patterns <strong>not</strong> express?</h3> + +<ul> + <li>Disjunctions of patterns, e.g., &ldquo;number or boolean&rdquo;. (Use a language non-terminal.)</li> + <li>Negations of patterns. (Compose <strong>not</strong> with <strong>redex-match?</strong>.)</li> + <li>Some non-regular patterns. (Named dots <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29"><code>..._N</code></a> or <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-language%29%29"><code>define-language</code></a> with a side condition may be able to help.)</li></ul> + +<h3 id="q-what-is-a-judgment">Q. What is a judgment?</h3> + +<p>A Redex <em>judgment</em> form defines a relation on terms. The relation is defined by a set of inference rules.</p> + +<p>Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:</p> + +<ul> + <li>well-formedness conditions (i.e., whether a term contains free variables)</li> + <li>type checking rules</li> + <li>type inference rules</li> + <li>evaluation relations</li></ul> + +<p>Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(define-judgment-form nat + #:mode (N= I I) + #:contract (N= N N) + [ + --- Zero= + (N= Zero Zero)] + [ + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1) + (N= N_0-- N_1--) + --- Plus1= + (N= N_0 N_1)])</code></pre> + +<ol> + <li>the language is <strong>nat</strong>; Redex uses the language to interpret patterns</li> + <li>the mode is <strong>(N= I I)</strong>; this means <strong>N=</strong> is the name of a judgment that expects two input terms (or, <strong>N=</strong> is a binary relation on terms)</li> + <li>the contract is <strong>(N= N N)</strong>; in other words, <strong>N=</strong> expects two terms that match the <strong>N</strong> non-terminal from the <strong>nat</strong> language</li> + <li>there are two inference rules, named <strong>Zero=</strong> and <strong>Plus1=</strong></li> + <li>the <strong>Zero=</strong> rule states that <strong>(N= Zero Zero)</strong> always holds</li> + <li>the <strong>Plus1=</strong> rule states that <strong>(N= N_0 N_1)</strong> holds if <strong>N_0</strong> and <strong>N_1</strong> are both <strong>Plus1</strong> terms whose contents are related by <strong>N=</strong></li></ol> + +<p>The <strong>where</strong> clauses are <em>guards</em>. When Redex tries to apply a rule with premises of the form <strong>(where pattern term)</strong>, it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-judgment-form%29%29">the Redex reference</a> for more.</p> + +<pre><code>(judgment-holds (N= Zero Zero)) +;; #true +(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Note: the inference rules form a <em>set</em>, not a <em>sequence</em>. So when you ask Redex whether <strong>(judgment-holds (N= Zero Zero))</strong>, it applies all rules that match <strong>(N= Zero Zero)</strong>. For <strong>N=</strong> this is just one rule, but in general it could be many rules.</p> + +<h3 id="q-what-is-a-judgment-form-mode">Q. What is a judgment form <strong>#:mode</strong>?</h3> + +<p>A <strong>#:mode</strong> declaration expects a list of the form <strong>(id pos-use &hellip;)</strong>, where <strong>id</strong> is an identifier and each <strong>pos-use</strong> is either <strong>I</strong> or <strong>O</strong>. These declarations say four things:</p> + +<ol> + <li><strong>id</strong> is the name of a new judgment form</li> + <li><strong>id</strong> expects <strong>N</strong> arguments, where <strong>N</strong> is the number of <strong>pos-use</strong> symbols</li> + <li><strong>id</strong> expects an <em>input</em> at each position where the mode contains an <strong>I</strong></li> + <li><strong>id</strong> produces an <em>output</em> at each position where the mode contains an <strong>O</strong></li></ol> + +<p>For example, a type inference judgment may take an expression as input and output a type. Here&rsquo;s a fast and easy type inference judgment for arithmetic expressions. Given any term <strong>e_0</strong>, the judgment outputs the type <strong>Int</strong>.</p> + +<pre><code>(define-language Arith + (e ::= integer (e + e)) + (τ ::= Int)) + +(define-judgment-form Arith + #:mode (infer-type I O) + #:contract (infer-type e τ) + [ + --- T-Int + (infer-type e_0 Int)])</code></pre> + +<h3 id="q-what-can-judgments-not-express">Q. What can judgments <strong>not</strong> express?</h3> + +<p>Redex does not support inference rules that require guessing.</p> + +<p>One example of this is a transitivity rule: "<strong>τ_0</strong> is related to <strong>τ_2</strong> if there exists a <strong>τ_1</strong> such that <strong>τ_0</strong> is related to <strong>τ_1</strong> and <strong>τ_1</strong> is related to <strong>τ_2</strong>". The following example tries to define a transitive subtyping (<strong>&lt;:</strong>) relation, but Redex rejects the <strong>S-Trans</strong> rule.</p> + +<pre><code>(define-language SomeTypes + (τ ::= (→ τ τ) Integer)) + +(define-judgment-form SomeTypes + #:mode (&lt;: I I) + #:contract (&lt;: τ τ) + [ + (&lt;: τ_0 τ_1) + (&lt;: τ_1 τ_2) + --- S-Trans + (&lt;: τ_0 τ_2)] + [ + --- S-Refl + (&lt;: τ_0 τ_0)] + [ + (&lt;: τ_dom-1 τ_dom-0) + (&lt;: τ_cod-0 τ_cod-1) + --- S-Arrow + (&lt;: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])</code></pre> + +<p>The error is that in the rule <strong>S-Trans</strong>, the named non-terminal <strong>τ_1</strong> appears in an input position but is not bound to a term.</p> + +<h3 id="q-what-is-a-metafunction">Q. What is a metafunction?</h3> + +<p>A metafunction is a term-level function on terms.</p> + +<p>Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.</p> + +<p>Here is a metafunction that returns <strong>#true</strong> when given two equal natural numbers. The definition is similar to the <strong>N=</strong> judgment form.</p> + +<pre><code>(define-metafunction nat + N=? : N N -&gt; boolean + [(N=? Zero Zero) + #true] + [(N=? N_0 N_1) + (N=? N_0-- N_1--) + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1)] + [(N=? N_0 N_1) + #false])</code></pre> + +<ul> + <li>the metafunction is named <strong>N=?</strong></li> + <li>its contract is <strong>N N -&gt; boolean</strong>, this means <strong>N=?</strong> expects 2 terms that match the <strong>N</strong> pattern and returns a term that matches the pattern <strong>boolean</strong></li> + <li>there are three cases; the second case is guarded by two <strong>where</strong> clauses</li></ul> + +<p>Any occurrence of <strong>(N=? &hellip;.)</strong> in any term is evaluated.</p> + +<pre><code>(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(term ((N=? Zero Zero) Zero)) +;; '(#true Zero) +(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Any occurrence of <strong>N=?</strong> outside a <strong>term</strong> is an error.</p> + +<p>Metafunction <strong>where</strong>-clauses are analogous to judgment form <strong>where</strong>-clauses. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._metafunction%29">the Redex reference</a> for more.</p> + +<p>Note: the cases in a metafunction form a <em>sequence</em>, not a <em>set</em>. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.</p> + +<h3 id="q-should-i-use-a-metafunction-or-a-judgment-form">Q. Should I use a metafunction or a judgment form?</h3> + +<p>Use a judgment form.</p> + +<p>Metafunctions are handy, but judgments are easier to read and debug and maintain.</p> + +<h3 id="q-what-is-a-reduction-relation">Q. What is a reduction relation?</h3> + +<p>A reduction relation is a set of term-rewriting rules.</p> + +<p>Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.</p> + +<ul> + <li>The language tells Redex how to interpret patterns.</li> + <li>The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.</li> + <li>The rewrite rules have the form <strong>(&mdash;&gt; term term guard &hellip;)</strong>. The term on the left is the input, the term on the right is the output.</li></ul> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._reduction-relation%29%29">the Redex reference</a> for a full description of the guards.</p> + +<p>The preferred way to define a reduction relation is to define a language that includes three non-terminals:</p> + +<ol> + <li>a non-terminal for the domain of the reduction relation</li> + <li>a non-terminal for a <em>subset</em> of the domain that cannot reduce further</li> + <li>a non-terminal for evaluation contexts</li></ol> + +<p>An evaluation context is a term that contains a <strong>hole</strong>. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite &mdash; in particular, the sub-term that matches the <strong>hole</strong>.</p> + +<p>In the following example, <strong>bexp</strong> is the domain of a reduction relation. A <strong>bexp</strong> term represents a boolean expression, which can be <strong>#true</strong> or <strong>#false</strong> or a conjunction of expressions or a disjunction of expressions. The boolean expressions <strong>#true</strong> and <strong>#false</strong> are also values (<strong>val</strong>); these cannot reduce further. The non-terminal <strong>E</strong> is for evaluation contexts.</p> + +<pre><code>(define-language Bool + (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp)) + (val ::= #true #false) + (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E))) + +(define step + (reduction-relation Bool + #:domain bexp + [--&gt; (in-hole E (val_lhs ∧ val_rhs)) + (in-hole E val_new) + ∧-step + (where val_new ,(and (term val_lhs) (term val_rhs)))] + [--&gt; (in-hole E (val_lhs ∨ val_rhs)) + (in-hole E val_new) + ∨-step + (where val_new ,(or (term val_lhs) (term val_rhs)))]))</code></pre> + +<p>The function <strong>apply-reduction-relation</strong> applies a reduction relation to a term and returns a list of ways that the term can step.</p> + +<pre><code>(apply-reduction-relation step (term #true)) +;; '() +(apply-reduction-relation step (term (#true ∧ #true))) +;; '(#true) +(apply-reduction-relation step (term (#true ∧ #false))) +;; '(#false) +(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true))) +;; '((#true ∧ #true))</code></pre> + +<p>Three things about the reduction relation <strong>step</strong>:</p> + +<ol> + <li>Using <strong>in-hole</strong> on the first argument of <strong>&mdash;&gt;</strong> searches a term for a subterm that Redex can apply a reduction rule to.</li> + <li>Using <strong>in-hole</strong> on the second argument of <strong>&mdash;&gt;</strong> puts a new value back into the <strong>hole</strong> in the evaluation context.</li> + <li>The unquote operator (<strong>,</strong>) escapes to &ldquo;Racket mode&rdquo; (see below) to evaluate a conjunction or disjunction.</li></ol> + +<p>A judgment or metafunction is a formal alternative to &ldquo;escaping to Racket&rdquo;, but escaping can be convenient.</p> + +<p>Note: the cases in a reduction relation form a <em>set</em>, not a <em>sequence</em>. If more than one case matches, Redex applies them all.</p> + +<h3 id="q-what-is-racket-mode-what-is-redex-mode">Q. What is &ldquo;Racket mode&rdquo;? What is &ldquo;Redex mode&rdquo;?</h3> + +<p>Code in a Redex model is sometimes evaluated in &ldquo;Racket mode&rdquo; and sometimes evaluated in &ldquo;Redex mode&rdquo;. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.</p> + +<p>Key points:</p> + +<ol> + <li>A Redex program starts in Racket mode.</li> + <li>The <strong>X</strong> in <strong>(term X)</strong> is evaluated in Redex mode &hellip;</li> + <li>&hellip; unless <strong>X</strong> contains unquoted sub-expressions. Unquoting escapes to Racket mode &hellip;</li> + <li>&hellip; and <strong>term</strong>s inside an unquoted sub-expression are evaluated in Redex mode.</li></ol> + +<p>In other words, <strong>term</strong> enters Redex mode and <strong>unquote</strong> (<strong>,</strong>) escapes back to Racket.</p> + +<p>Redex implicitly switches to Redex mode in a few other places, for example:</p> + +<ul> + <li>the right-side of a <strong>where</strong> clause is in Redex mode</li> + <li>the result of a metafunction is in Redex mode</li></ul> + +<p>When in doubt, try using an <strong>unquote</strong>. Redex will raise an exception if it finds an unquote in Racket mode.</p> + +<h3 id="q-are-side-conditions-evaluated-in-racket-mode-or-redex-mode">Q. Are <strong>side-condition</strong>s evaluated in &ldquo;Racket mode&rdquo; or &ldquo;Redex mode&rdquo;?</h3> + +<p>A <strong>(side-condition e)</strong> sometimes evaluates <strong>e</strong> as a Racket expression and sometimes evaluates <strong>e</strong> as a Redex expression.</p> + +<ul> + <li>reduction relations and metafunctions expect a <strong>Racket</strong> expression</li> + <li>judgments expect a <strong>Redex</strong> expression</li></ul> + +<h3 id="q-what-is-a-binding-form">Q. What is a binding form?</h3> + +<p>In the lambda calculus, <strong>λ</strong>-terms bind variables. A term <strong>(λ x M)</strong> means that any free occurrence of <strong>x</strong> in the sub-term <strong>M</strong> refers to the <strong>x</strong> from the <strong>λ</strong>-term.</p> + +<p>Redex can express this idea with a binding form.</p> + +<pre><code>(define-language Λ + [e ::= (e e) x (λ x e)] + [x ::= variable-not-otherwise-mentioned] + #:binding-forms + (λ x_0 e_0 #:refers-to x_0))</code></pre> + +<p>Note: all the non-terminals in a language must be defined before the <strong>#:binding-forms</strong> keyword. If a non-terminal definition appears after the <strong>#:binding-forms</strong> keyword, then Redex will interpret the &ldquo;definition&rdquo; as a binding form.</p> + +<p>Binding forms work together with Redex&rsquo;s functions for substitution and alphabetic equivalence.</p> + +<pre><code>(alpha-equivalent? Λ + (term (λ x x)) + (term (λ y y)))) +;; #true + +(define-metafunction Λ + test-substitute : e -&gt; e + [(test-substitute (λ x_0 e_0)) + (substitute e_0 x_0 y)]) +(term (test-substitute (λ z (z z)))) +;; '(y y)</code></pre> + +<h3 id="q-what-is--what-is-">Q. What is <strong>&hellip;</strong>? What is <strong>&hellip;.</strong>?</h3> + +<p>Three dots (<strong>&hellip;</strong>) is for building patterns. If <strong>p</strong> is a pattern then <strong>(p &hellip;)</strong> matches any list whose elements all match <strong>p</strong>.</p> + +<pre><code>(define-language L) +(redex-match? L (number ... boolean ...) (term (1 2 #true #true))) +;; #true</code></pre> + +<p>Four dots (<strong>&hellip;.</strong>) may be used in <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-extended-language%29%29"><strong>define-extended-language</strong></a> to extend a previously-defined non-terminal.</p> + +<pre><code>(define-language C + (keyword ::= auto break case)) +(define-extended-language C++ + C + (keyword ::= .... class)) + +(redex-match? C keyword (term auto)) +;; #true +(redex-match? C keyword (term class)) +;; #false +(redex-match? C++ keyword (term auto)) +;; #true +(redex-match? C++ keyword (term class)) +;; #true</code></pre> + +<h3 id="q-where-to-learn-more-about-redex">Q. Where to learn more about Redex?</h3> + +<p>&ldquo;Critical path&rdquo; resources:</p> + +<ul> + <li>Code examples for this post: <a href="https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt">https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt</a></li> + <li>Redex documentation: <a href="http://docs.racket-lang.org/redex/index.html">http://docs.racket-lang.org/redex/index.html</a></li> + <li><a href="https://dvanhorn.github.io/redex-aam-tutorial/"><em>An Introduction to Redex with Abstracting Abstract Machines</em></a> by David Van Horn</li> + <li><a href="https://www.leafac.com/playing-the-game-with-plt-redex/"><em>Playing the Game with PLT Redex</em></a> by Leandro Facchinetti</li> + <li><a href="https://williamjbowman.com/doc/experimenting-with-redex/index.html"><em>Experimenting with Languages in Redex</em></a> by William J. Bowman</li></ul> + +<p>&ldquo;Procrastination&rdquo; resources:</p> + +<ul> + <li><a href="http://tata.gforge.inria.fr/"><em>Tree Automata Techniques and Applications</em></a></li> + <li><a href="http://lamport.azurewebsites.net/pubs/lamport-types.pdf"><em>Should your Specification Language be Typed?</em></a></li> + <li>Redex source code (see <code>redex-lib/</code>): <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=Author-Sam-Caldwell&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Fall 2016 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/?utm_source=Author-Sam-Caldwell&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + Mon, 02 Jan 2017 16:39:37 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Stephen-Chang.atom.xml b/blog/feeds/Author-Stephen-Chang.atom.xml new file mode 100644 index 00000000..efbfe12c --- /dev/null +++ b/blog/feeds/Author-Stephen-Chang.atom.xml @@ -0,0 +1,20 @@ + + + PRL Blog: Posts tagged 'Author: Stephen Chang' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Stephen-Chang-html + 2018-11-30T14:55:30Z + + Turnstile Mailing List + + urn:http-prl-ccs-neu-edu:-blog-2018-11-30-turnstile-mailing-list + 2018-11-30T14:55:30Z + 2018-11-30T14:55:30Z + + Stephen Chang + +<p><a href="https://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> now has a mailing list: <a href="https://groups.google.com/forum/#!forum/turnstile-users">https://groups.google.com/forum/#!forum/turnstile-users</a></p> +<!-- more--> + +<p>There is also a <code>#turnstile</code> channel on <a href="https://racket.slack.com">the Racket Slack</a>.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Stephen-Chang.rss.xml b/blog/feeds/Author-Stephen-Chang.rss.xml new file mode 100644 index 00000000..cbf65d8d --- /dev/null +++ b/blog/feeds/Author-Stephen-Chang.rss.xml @@ -0,0 +1,20 @@ + + + + PRL Blog: Posts tagged 'Author: Stephen Chang' + PRL Blog: Posts tagged 'Author: Stephen Chang' + http://prl.ccs.neu.edu/blog/tags/Author-Stephen-Chang.html + Fri, 30 Nov 2018 14:55:30 UT + Fri, 30 Nov 2018 14:55:30 UT + 1800 + + Turnstile Mailing List + http://prl.ccs.neu.edu/blog/2018/11/30/turnstile-mailing-list/?utm_source=Author-Stephen-Chang&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-11-30-turnstile-mailing-list + Fri, 30 Nov 2018 14:55:30 UT + Stephen Chang + +<p><a href="https://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> now has a mailing list: <a href="https://groups.google.com/forum/#!forum/turnstile-users">https://groups.google.com/forum/#!forum/turnstile-users</a></p> +<!-- more--> + +<p>There is also a <code>#turnstile</code> channel on <a href="https://racket.slack.com">the Racket Slack</a>.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Tony-Garnock-Jones.atom.xml b/blog/feeds/Author-Tony-Garnock-Jones.atom.xml new file mode 100644 index 00000000..d9c54468 --- /dev/null +++ b/blog/feeds/Author-Tony-Garnock-Jones.atom.xml @@ -0,0 +1,65 @@ + + + PRL Blog: Posts tagged 'Author: Tony Garnock-Jones' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Tony-Garnock-Jones-html + 2019-05-11T00:03:16Z + + [Conversational Concurrency (cross-post)](https://eighty-twenty.org/2018/01/24/conversational-concurrency) + + urn:http-prl-ccs-neu-edu:-blog-2019-05-11-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency + 2019-05-11T00:03:16Z + 2019-05-11T00:03:16Z + + Tony Garnock-Jones + + + Conversational Context and Concurrency + + urn:http-prl-ccs-neu-edu:-blog-2017-02-15-conversational-context-and-concurrency + 2017-02-15T01:21:55Z + 2017-02-15T01:21:55Z + + Tony Garnock-Jones + <!-- more--> + +<p>When programs are written with concurrency in mind, the programmer reasons about the interactions between concurrent components or agents in the program. This includes exchange of information, as well as management of resources, handling of partial failure, collective decision-making and so on.</p> + +<p>These components might be objects, or threads, or processes, or actors, or some more nebulous and loosely-defined concept; a group of callbacks, perhaps. The programmer has the notion of an agent in their mind, which translates into some representation of that agent in the program.</p> + +<p>We think about the contexts (because there can be more than one) in which agents exist in two different ways. From each agent&rsquo;s perspective, the important thing to think about is the boundary between the agent and everything else in the system. But from the system perspective, we often think about <em>conversations</em> between agents, whether it&rsquo;s just two having an exchange, or a whole group collaborating on some task. Agents in a conversation play different roles, join and leave the group, and build shared conversational state.</p> + +<p>In this talk, I used the idea of these <em>conversational contexts</em> as a lens through which to view the development of various metaphors and mechanisms of communication and coordination. I presented four <em>computational models</em> for concurrent interaction:</p> + +<ul> + <li>monitors, and shared memory concurrency generally</li> + <li>the actor model</li> + <li>channel-based communication</li> + <li>tuplespaces</li></ul> + +<p>These aren&rsquo;t full programming languages, but there are many <em>programming models</em> that build upon them. In some cases, development of these ideas has progressed all the way up to <em>system models</em> including user interaction and so forth.</p> + +<p>The linked lecture notes include informal sketches of reduction semantics for each of the four models, plus a handful of small examples to give a feel for them.</p> + +<p>Lecture Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/conversational-context-and-concurrency/index.md">https://github.com/nuprl/hopl-s2017/tree/master/conversational-context-and-concurrency/index.md</a></li></ul> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-01-31.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;01&ndash;31.md</a></li></ul> + + History of Actors + + urn:http-prl-ccs-neu-edu:-blog-2016-10-19-history-of-actors + 2016-10-19T17:26:16Z + 2016-10-19T17:26:16Z + + Tony Garnock-Jones + +<p>Christos Dimoulas is currently teaching a <a href="http://www.seas.harvard.edu/courses/cs252/2016fa/">&ldquo;History of Programming Languages&rdquo; class at Harvard</a>. The class is, as Christos writes, &ldquo;definitely not about <a href="https://www.levenez.com/lang/lang_letter.pdf">this</a>&rdquo;; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.</p> + +<p>On Monday, I presented &ldquo;the History of Actors&rdquo; for the class. I&rsquo;ve made the written-out talk notes and an annotated bibliography available <a href="https://eighty-twenty.org/2016/10/18/actors-hopl">here</a>.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Tony-Garnock-Jones.rss.xml b/blog/feeds/Author-Tony-Garnock-Jones.rss.xml new file mode 100644 index 00000000..a2cc07ee --- /dev/null +++ b/blog/feeds/Author-Tony-Garnock-Jones.rss.xml @@ -0,0 +1,61 @@ + + + + PRL Blog: Posts tagged 'Author: Tony Garnock-Jones' + PRL Blog: Posts tagged 'Author: Tony Garnock-Jones' + http://prl.ccs.neu.edu/blog/tags/Author-Tony-Garnock-Jones.html + Sat, 11 May 2019 00:03:16 UT + Sat, 11 May 2019 00:03:16 UT + 1800 + + [Conversational Concurrency (cross-post)](https://eighty-twenty.org/2018/01/24/conversational-concurrency) + http://prl.ccs.neu.edu/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/?utm_source=Author-Tony-Garnock-Jones&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-05-11-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency + Sat, 11 May 2019 00:03:16 UT + Tony Garnock-Jones + + + Conversational Context and Concurrency + http://prl.ccs.neu.edu/blog/2017/02/15/conversational-context-and-concurrency/?utm_source=Author-Tony-Garnock-Jones&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-15-conversational-context-and-concurrency + Wed, 15 Feb 2017 01:21:55 UT + Tony Garnock-Jones + <!-- more--> + +<p>When programs are written with concurrency in mind, the programmer reasons about the interactions between concurrent components or agents in the program. This includes exchange of information, as well as management of resources, handling of partial failure, collective decision-making and so on.</p> + +<p>These components might be objects, or threads, or processes, or actors, or some more nebulous and loosely-defined concept; a group of callbacks, perhaps. The programmer has the notion of an agent in their mind, which translates into some representation of that agent in the program.</p> + +<p>We think about the contexts (because there can be more than one) in which agents exist in two different ways. From each agent&rsquo;s perspective, the important thing to think about is the boundary between the agent and everything else in the system. But from the system perspective, we often think about <em>conversations</em> between agents, whether it&rsquo;s just two having an exchange, or a whole group collaborating on some task. Agents in a conversation play different roles, join and leave the group, and build shared conversational state.</p> + +<p>In this talk, I used the idea of these <em>conversational contexts</em> as a lens through which to view the development of various metaphors and mechanisms of communication and coordination. I presented four <em>computational models</em> for concurrent interaction:</p> + +<ul> + <li>monitors, and shared memory concurrency generally</li> + <li>the actor model</li> + <li>channel-based communication</li> + <li>tuplespaces</li></ul> + +<p>These aren&rsquo;t full programming languages, but there are many <em>programming models</em> that build upon them. In some cases, development of these ideas has progressed all the way up to <em>system models</em> including user interaction and so forth.</p> + +<p>The linked lecture notes include informal sketches of reduction semantics for each of the four models, plus a handful of small examples to give a feel for them.</p> + +<p>Lecture Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/conversational-context-and-concurrency/index.md">https://github.com/nuprl/hopl-s2017/tree/master/conversational-context-and-concurrency/index.md</a></li></ul> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-01-31.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;01&ndash;31.md</a></li></ul> + + History of Actors + http://prl.ccs.neu.edu/blog/2016/10/19/history-of-actors/?utm_source=Author-Tony-Garnock-Jones&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-19-history-of-actors + Wed, 19 Oct 2016 17:26:16 UT + Tony Garnock-Jones + +<p>Christos Dimoulas is currently teaching a <a href="http://www.seas.harvard.edu/courses/cs252/2016fa/">&ldquo;History of Programming Languages&rdquo; class at Harvard</a>. The class is, as Christos writes, &ldquo;definitely not about <a href="https://www.levenez.com/lang/lang_letter.pdf">this</a>&rdquo;; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.</p> + +<p>On Monday, I presented &ldquo;the History of Actors&rdquo; for the class. I&rsquo;ve made the written-out talk notes and an annotated bibliography available <a href="https://eighty-twenty.org/2016/10/18/actors-hopl">here</a>.</p> \ No newline at end of file diff --git a/blog/feeds/Author-William-J-Bowman.atom.xml b/blog/feeds/Author-William-J-Bowman.atom.xml new file mode 100644 index 00000000..68804652 --- /dev/null +++ b/blog/feeds/Author-William-J-Bowman.atom.xml @@ -0,0 +1,77 @@ + + + PRL Blog: Posts tagged 'Author: William J. Bowman' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-William-J-Bowman-html + 2018-01-19T17:05:00Z + + [Untyped Programs Don't Exist (cross-post)](https://williamjbowman.com/blog/2018/01/19/untyped-programs-don-t-exist/) + + urn:http-prl-ccs-neu-edu:-blog-2018-01-19-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist + 2018-01-19T17:05:00Z + 2018-01-19T17:05:00Z + + William J. Bowman + + + [Why am I going to ICFP 2017? (cross-post)](https://williamjbowman.com/blog/2017/08/29/why-am-i-going-to-icfp-2017/) + + urn:http-prl-ccs-neu-edu:-blog-2017-08-29-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017 + 2017-08-29T13:33:21Z + 2017-08-29T13:33:21Z + + William J. Bowman + + + Type-Directed Compilation, Parts I and II + + urn:http-prl-ccs-neu-edu:-blog-2017-04-17-type-directed-compilation-parts-i-and-ii + 2017-04-17T12:00:17Z + 2017-04-17T12:00:17Z + Leif Andersen + William J. Bowman + + Leif Andersen, William J. Bowman + <!-- more--> + +<h3 id="part-i-type-directed-compilation-by-leif-andersen">Part I: <em>Type-Directed Compilation</em>, by Leif Andersen.</h3> + +<p>In this talk we discuss the history of type directed compilation. We start with Xavier Leroy&rsquo;s seminal paper: <a href="http://gallium.inria.fr/~xleroy/publi/unboxed-polymorphism.pdf"><em>Unboxed Objects and Polymorphic Typing</em></a>, continue to <a href="https://www.cs.cmu.edu/~rwh/papers/til/pldi96.pdf">TIL</a> (Typed Intermediate Language), and finish up with <a href="https://dash.harvard.edu/handle/1/2797451">TAL</a> (Typed Assembly Language). We talk about what it means for a compiler to be typed preserving, and give examples of optimizations that are enabled by types.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-24.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;24.md</a></li></ul> + +<h3 id="part-ii-dependent-type-directed-compilation-by-william-j-bowman">Part II: <em>Dependent Type-Directed Compilation</em>, by William J. Bowman</h3> + +<p>A certifying compiler is not verified, but it produces a proof of correctness for each binary. This proof can be independently checked to show that the binary was compiled correctly, removing the compiler from the trusted code base. Certifying compilation has its roots in preserving type-preserving compilation, and in particular in preserving dependent types. We start the history of dependent-type-preserving compilation with a compiler from C to Assembly. We&rsquo;ll see a result showing that preserving dependent types isn&rsquo;t possible, and then we&rsquo;ll do it anyway.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-28.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;28.md</a></li></ul> + +<p>Notes (to appear here, eventually):</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation">https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation</a></li></ul> + + [What even is compiler correctness? (cross-post)](https://williamjbowman.com/blog/2017/03/24/what-even-is-compiler-correctness/) + + urn:http-prl-ccs-neu-edu:-blog-2017-03-24-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness + 2017-03-24T23:11:17Z + 2017-03-24T23:11:17Z + + William J. Bowman + + + [Toward Type-Preserving Compilation of Coq, at POPL17 SRC (cross-post)](https://williamjbowman.com/blog/2017/01/03/toward-type-preserving-compilation-of-coq-at-popl17-src/) + + urn:http-prl-ccs-neu-edu:-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src + 2017-01-03T15:15:57Z + 2017-01-03T15:15:57Z + + William J. Bowman + \ No newline at end of file diff --git a/blog/feeds/Author-William-J-Bowman.rss.xml b/blog/feeds/Author-William-J-Bowman.rss.xml new file mode 100644 index 00000000..0d0a6cd1 --- /dev/null +++ b/blog/feeds/Author-William-J-Bowman.rss.xml @@ -0,0 +1,67 @@ + + + + PRL Blog: Posts tagged 'Author: William J. Bowman' + PRL Blog: Posts tagged 'Author: William J. Bowman' + http://prl.ccs.neu.edu/blog/tags/Author-William-J-Bowman.html + Fri, 19 Jan 2018 17:05:00 UT + Fri, 19 Jan 2018 17:05:00 UT + 1800 + + [Untyped Programs Don't Exist (cross-post)](https://williamjbowman.com/blog/2018/01/19/untyped-programs-don-t-exist/) + http://prl.ccs.neu.edu/blog/2018/01/19/-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist/?utm_source=Author-William-J-Bowman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-01-19-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist + Fri, 19 Jan 2018 17:05:00 UT + William J. Bowman + + + [Why am I going to ICFP 2017? (cross-post)](https://williamjbowman.com/blog/2017/08/29/why-am-i-going-to-icfp-2017/) + http://prl.ccs.neu.edu/blog/2017/08/29/-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017/?utm_source=Author-William-J-Bowman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-29-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017 + Tue, 29 Aug 2017 13:33:21 UT + William J. Bowman + + + Type-Directed Compilation, Parts I and II + http://prl.ccs.neu.edu/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/?utm_source=Author-William-J-Bowman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-04-17-type-directed-compilation-parts-i-and-ii + Mon, 17 Apr 2017 12:00:17 UT + Leif Andersen, William J. Bowman + <!-- more--> + +<h3 id="part-i-type-directed-compilation-by-leif-andersen">Part I: <em>Type-Directed Compilation</em>, by Leif Andersen.</h3> + +<p>In this talk we discuss the history of type directed compilation. We start with Xavier Leroy&rsquo;s seminal paper: <a href="http://gallium.inria.fr/~xleroy/publi/unboxed-polymorphism.pdf"><em>Unboxed Objects and Polymorphic Typing</em></a>, continue to <a href="https://www.cs.cmu.edu/~rwh/papers/til/pldi96.pdf">TIL</a> (Typed Intermediate Language), and finish up with <a href="https://dash.harvard.edu/handle/1/2797451">TAL</a> (Typed Assembly Language). We talk about what it means for a compiler to be typed preserving, and give examples of optimizations that are enabled by types.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-24.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;24.md</a></li></ul> + +<h3 id="part-ii-dependent-type-directed-compilation-by-william-j-bowman">Part II: <em>Dependent Type-Directed Compilation</em>, by William J. Bowman</h3> + +<p>A certifying compiler is not verified, but it produces a proof of correctness for each binary. This proof can be independently checked to show that the binary was compiled correctly, removing the compiler from the trusted code base. Certifying compilation has its roots in preserving type-preserving compilation, and in particular in preserving dependent types. We start the history of dependent-type-preserving compilation with a compiler from C to Assembly. We&rsquo;ll see a result showing that preserving dependent types isn&rsquo;t possible, and then we&rsquo;ll do it anyway.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-28.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;28.md</a></li></ul> + +<p>Notes (to appear here, eventually):</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation">https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation</a></li></ul> + + [What even is compiler correctness? (cross-post)](https://williamjbowman.com/blog/2017/03/24/what-even-is-compiler-correctness/) + http://prl.ccs.neu.edu/blog/2017/03/24/-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness/?utm_source=Author-William-J-Bowman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-03-24-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness + Fri, 24 Mar 2017 23:11:17 UT + William J. Bowman + + + [Toward Type-Preserving Compilation of Coq, at POPL17 SRC (cross-post)](https://williamjbowman.com/blog/2017/01/03/toward-type-preserving-compilation-of-coq-at-popl17-src/) + http://prl.ccs.neu.edu/blog/2017/01/03/-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src/?utm_source=Author-William-J-Bowman&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src + Tue, 03 Jan 2017 15:15:57 UT + William J. Bowman + \ No newline at end of file diff --git a/blog/feeds/Author-Zeina-Migeed.atom.xml b/blog/feeds/Author-Zeina-Migeed.atom.xml new file mode 100644 index 00000000..65652b16 --- /dev/null +++ b/blog/feeds/Author-Zeina-Migeed.atom.xml @@ -0,0 +1,133 @@ + + + PRL Blog: Posts tagged 'Author: Zeina Migeed' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Author-Zeina-Migeed-html + 2018-05-08T15:37:37Z + + Sampling Gradual Typing Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + 2018-05-08T15:37:37Z + 2018-05-08T15:37:37Z + Ben Greenman + Zeina Migeed + + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> \ No newline at end of file diff --git a/blog/feeds/Author-Zeina-Migeed.rss.xml b/blog/feeds/Author-Zeina-Migeed.rss.xml new file mode 100644 index 00000000..5544d133 --- /dev/null +++ b/blog/feeds/Author-Zeina-Migeed.rss.xml @@ -0,0 +1,131 @@ + + + + PRL Blog: Posts tagged 'Author: Zeina Migeed' + PRL Blog: Posts tagged 'Author: Zeina Migeed' + http://prl.ccs.neu.edu/blog/tags/Author-Zeina-Migeed.html + Tue, 08 May 2018 15:37:37 UT + Tue, 08 May 2018 15:37:37 UT + 1800 + + Sampling Gradual Typing Performance + http://prl.ccs.neu.edu/blog/2018/05/08/sampling-gradual-typing-performance/?utm_source=Author-Zeina-Migeed&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + Tue, 08 May 2018 15:37:37 UT + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> \ No newline at end of file diff --git a/blog/feeds/CRDTs.atom.xml b/blog/feeds/CRDTs.atom.xml new file mode 100644 index 00000000..699bd2ee --- /dev/null +++ b/blog/feeds/CRDTs.atom.xml @@ -0,0 +1,144 @@ + + + PRL Blog: Posts tagged 'CRDTs' + + + urn:http-prl-ccs-neu-edu:-blog-tags-CRDTs-html + 2017-10-22T11:59:06Z + + Monotonicity Types: Towards A Type System for Eventual Consistency + + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + 2017-10-22T11:59:06Z + 2017-10-22T11:59:06Z + + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> \ No newline at end of file diff --git a/blog/feeds/CRDTs.rss.xml b/blog/feeds/CRDTs.rss.xml new file mode 100644 index 00000000..349ca55d --- /dev/null +++ b/blog/feeds/CRDTs.rss.xml @@ -0,0 +1,144 @@ + + + + PRL Blog: Posts tagged 'CRDTs' + PRL Blog: Posts tagged 'CRDTs' + http://prl.ccs.neu.edu/blog/tags/CRDTs.html + Sun, 22 Oct 2017 11:59:06 UT + Sun, 22 Oct 2017 11:59:06 UT + 1800 + + Monotonicity Types: Towards A Type System for Eventual Consistency + http://prl.ccs.neu.edu/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/?utm_source=CRDTs&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + Sun, 22 Oct 2017 11:59:06 UT + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> \ No newline at end of file diff --git a/blog/feeds/Emacs.atom.xml b/blog/feeds/Emacs.atom.xml new file mode 100644 index 00000000..41cf1c04 --- /dev/null +++ b/blog/feeds/Emacs.atom.xml @@ -0,0 +1,70 @@ + + + PRL Blog: Posts tagged 'Emacs' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Emacs-html + 2016-10-17T21:48:25Z + + Emacs daemon for fast editor startup + + urn:http-prl-ccs-neu-edu:-blog-2016-10-17-emacs-daemon-for-fast-editor-startup + 2016-10-17T21:48:25Z + 2016-10-17T21:48:25Z + + Gabriel Scherer + +<p>In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs&rsquo; bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors &mdash; on each git commit for example, or when I use a <a href="https://addons.mozilla.org/en-US/firefox/addon/its-all-text/">Firefox extension</a> to edit a textarea content in a proper editor.</p> + +<p>In this blog post, I describe how to use <code>emacsclient</code> to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.</p> +<!-- more--> + +<p>Emacs has long supported a client/server mode: a daemon emacs instance is loaded in the background, and whenever you request a new emacs window you can creates a new frame (~ window) using the same instance. This means that the startup time is dramatically reduced after the daemon is launched, as for example the execution of your personal configuration code does not have to be repeated.</p> + +<p>To use it, I have this code as <code>/usr/bin/editor</code>:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="ch">#!/bin/bash</span> +emacsclient -a <span class="s2">""</span> -c <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The empty <code>-a</code> parameter means that if no daemon exists, it should start one in the background and retry. The <code>-c</code> option means that a new frame (window) should be created instead of reusing an existing one. <code>"$@"</code>means that when the script is invoked with a path as command-line parameter (<code>editor /tmp/foo.txt</code>), the corresponding file will be opened.</p> + +<p>Finally, my <code>.bash_profile</code> sets the <code>EDITOR</code> variable to <code>editor</code> (<code>export EDITOR=/usr/bin/editor</code>); this environment variable is what most tools (git included) will use to invoke a text editor.</p> + +<p>On my machine, starting the daemon takes 1.4s. Creating a client windows takes around 0.150s.</p> + +<p>If you want to control the environment in which the daemon process is started, you can launch it explicitly by running <code>emacs --daemon</code>.</p> + +<p>Cool kids use <a href="http://spacemacs.org/">Spacemacs</a> these days, which comes with all sort of convenient settings built in, and I&rsquo;m told that it does daemonization out of the box. I haven&rsquo;t taken the time to give Spacemacs a try yet.</p> + +<p>Finally, sometimes having all editor windows share the same process is not the right thing, because I do stuff which makes Emacs a bit unstable. (It&rsquo;s not very good at asynchronous communication with the rest of the world, so for example accessing a file through SSH from Emacs can hang the process when network goes bad.). I&rsquo;ve been bitten a few times by a crash of all editor windows at the same time, and since then, when I know I&rsquo;m going to do &ldquo;heavy stuff&rdquo;, I launch a separate process for it (just <code>emacs</code> instead of <code>editor</code> or <code>emacsclient</code>).</p> + +<p>P.S.: To precisely measure the startup time, ask Emacs to evaluate a Lisp expression on startup, to kill it immediately.:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span>$ <span class="nb">time</span> emacs --eval <span class="s2">"(save-buffers-kill-terminal)"</span> +$ <span class="nb">time</span> emacsclient -a <span class="s1">&#39;&#39;</span> -c -e <span class="s2">"(save-buffers-kill-terminal)"</span> +</pre></div> +</td></tr></tbody></table> +</div> \ No newline at end of file diff --git a/blog/feeds/Emacs.rss.xml b/blog/feeds/Emacs.rss.xml new file mode 100644 index 00000000..5a6a1efa --- /dev/null +++ b/blog/feeds/Emacs.rss.xml @@ -0,0 +1,70 @@ + + + + PRL Blog: Posts tagged 'Emacs' + PRL Blog: Posts tagged 'Emacs' + http://prl.ccs.neu.edu/blog/tags/Emacs.html + Mon, 17 Oct 2016 21:48:25 UT + Mon, 17 Oct 2016 21:48:25 UT + 1800 + + Emacs daemon for fast editor startup + http://prl.ccs.neu.edu/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/?utm_source=Emacs&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-17-emacs-daemon-for-fast-editor-startup + Mon, 17 Oct 2016 21:48:25 UT + Gabriel Scherer + +<p>In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs&rsquo; bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors &mdash; on each git commit for example, or when I use a <a href="https://addons.mozilla.org/en-US/firefox/addon/its-all-text/">Firefox extension</a> to edit a textarea content in a proper editor.</p> + +<p>In this blog post, I describe how to use <code>emacsclient</code> to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.</p> +<!-- more--> + +<p>Emacs has long supported a client/server mode: a daemon emacs instance is loaded in the background, and whenever you request a new emacs window you can creates a new frame (~ window) using the same instance. This means that the startup time is dramatically reduced after the daemon is launched, as for example the execution of your personal configuration code does not have to be repeated.</p> + +<p>To use it, I have this code as <code>/usr/bin/editor</code>:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="ch">#!/bin/bash</span> +emacsclient -a <span class="s2">""</span> -c <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The empty <code>-a</code> parameter means that if no daemon exists, it should start one in the background and retry. The <code>-c</code> option means that a new frame (window) should be created instead of reusing an existing one. <code>"$@"</code>means that when the script is invoked with a path as command-line parameter (<code>editor /tmp/foo.txt</code>), the corresponding file will be opened.</p> + +<p>Finally, my <code>.bash_profile</code> sets the <code>EDITOR</code> variable to <code>editor</code> (<code>export EDITOR=/usr/bin/editor</code>); this environment variable is what most tools (git included) will use to invoke a text editor.</p> + +<p>On my machine, starting the daemon takes 1.4s. Creating a client windows takes around 0.150s.</p> + +<p>If you want to control the environment in which the daemon process is started, you can launch it explicitly by running <code>emacs --daemon</code>.</p> + +<p>Cool kids use <a href="http://spacemacs.org/">Spacemacs</a> these days, which comes with all sort of convenient settings built in, and I&rsquo;m told that it does daemonization out of the box. I haven&rsquo;t taken the time to give Spacemacs a try yet.</p> + +<p>Finally, sometimes having all editor windows share the same process is not the right thing, because I do stuff which makes Emacs a bit unstable. (It&rsquo;s not very good at asynchronous communication with the rest of the world, so for example accessing a file through SSH from Emacs can hang the process when network goes bad.). I&rsquo;ve been bitten a few times by a crash of all editor windows at the same time, and since then, when I know I&rsquo;m going to do &ldquo;heavy stuff&rdquo;, I launch a separate process for it (just <code>emacs</code> instead of <code>editor</code> or <code>emacsclient</code>).</p> + +<p>P.S.: To precisely measure the startup time, ask Emacs to evaluate a Lisp expression on startup, to kill it immediately.:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span>$ <span class="nb">time</span> emacs --eval <span class="s2">"(save-buffers-kill-terminal)"</span> +$ <span class="nb">time</span> emacsclient -a <span class="s1">&#39;&#39;</span> -c -e <span class="s2">"(save-buffers-kill-terminal)"</span> +</pre></div> +</td></tr></tbody></table> +</div> \ No newline at end of file diff --git a/blog/feeds/FFI.atom.xml b/blog/feeds/FFI.atom.xml new file mode 100644 index 00000000..9534a5f8 --- /dev/null +++ b/blog/feeds/FFI.atom.xml @@ -0,0 +1,2203 @@ + + + PRL Blog: Posts tagged 'FFI' + + + urn:http-prl-ccs-neu-edu:-blog-tags-FFI-html + 2016-07-11T17:33:40Z + + Tutorial: Racket FFI, part 3 + + urn:http-prl-ccs-neu-edu:-blog-2016-07-11-tutorial-racket-ffi-part-3 + 2016-07-11T17:33:40Z + 2016-07-11T17:33:40Z + + Asumu Takikawa + +<p>This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a> +and part 2 +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a>.</p> + +<p>In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.</p> +<!--more--> + +<p>As in the second post, let&rsquo;s start with some prologue code that establishes +the definitions from the previous two posts. But first, I&rsquo;m getting tired of +writing the <span class="stt">#:c-id identifier</span> notation for the underscored C function +names.</p> + +<p>Instead, let&rsquo;s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">$</span><span class="hspace">&nbsp;</span><span class="RktMeta">raco</span><span class="hspace">&nbsp;</span><span class="RktMeta">pkg</span><span class="hspace">&nbsp;</span><span class="RktMeta">install</span><span class="hspace">&nbsp;</span><span class="RktMeta">ffi-definer-convention</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>or you can just execute the following snippet in Racket:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pkg</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">pkg-install-command</span><span class="hspace">&nbsp;</span><span class="RktPn">#:skip-installed</span><span class="hspace">&nbsp;</span><span class="RktVal">#t</span><span class="hspace">&nbsp;</span><span class="RktVal">"ffi-definer-convention"</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Downloading repository git://github.com/takikawa/racket-ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: version: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: platform: x86_64-linux [3m]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: target machine: racket</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: installation name: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: variants: 3m</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main collects: /usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: collects paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main pkgs: /usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: pkgs paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: links files: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main docs: /usr/share/racket/doc</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- updating info-domain tables ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- pre-installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing foreign libraries --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing shared files ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- compiling collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- parallel build using 4 jobs ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 making: &lt;pkgs&gt;/ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- creating launchers --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing man pages --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- building documentation --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 running: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 rendering: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 2 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/local-redirect.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 1 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/release.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/search.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/start.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- post-installing collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This will install the package and compile its contents. If you&rsquo;re curious, the +docs for the package are available +<a href="http://docs.racket-lang.org/ffi-definer-convention/index.html">here</a>.</p> + +<p><span style="font-weight: bold">Note:</span> if you&rsquo;ve never installed a package before, you may want to glance +at the +<a href="http://docs.racket-lang.org/pkg/getting-started.html">package system docs</a>. +A tl;dr of packages is that they bundle Racket <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/collects.html#%28tech._collection%29"><span class="techinside">collections</span></a>, which are +sets of modules that you can refer to in a location-independent fashion such as +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> or <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28mod-path._racket%2Flist%29"><span class="RktSym">racket/list</span></a></span>.</p> + +<p>Anyhow, here&rsquo;s the prologue code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">avoid conflict with below</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._except-in%29%29">except-in</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">define-ffi-definer</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the new 3rd-party pkg</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi-definer-convention</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">describes how to transform from</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Racket to C ids</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:make-c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">convention:hyphen-&gt;underscore</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">note lack of #:c-id keyword arguments</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">(_cairo_t -&gt; Void) -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">do some drawing and give us the pict</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">f</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">f</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Notice that the <span class="RktWrap"><span class="RktSym">define-cairo</span></span> forms don&rsquo;t have any <span class="stt">#:c-id</span> keywords +anymore. Instead, the prologue code uses an overriden <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> +from my package that supports a <span class="stt">#:make-c-id</span> keyword that lets you specify +a naming convention to follow.</p> + +<p>Also, instead of creating a single bitmap and drawing into it, we now have a +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> function that takes a drawing function. When called, +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> will call the given function with a new bitmap object and +return the result.</p> + +<p>Now let&rsquo;s get to the main point of this blog post. Let&rsquo;s say that we want to play +with Cairo <a href="https://www.cairographics.org/manual/cairo-Paths.html">path</a> +objects this time. A path is +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-t">defined</a> +as a struct with the following structure:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_status_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">status</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it&rsquo;s +useful to define C types for the types of values in the path struct&rsquo;s fields. First, +let&rsquo;s specify that a <span class="stt">cairo_status_t</span> is an integer type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>It&rsquo;s actually an enum, but for the examples in this post we don&rsquo;t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-data-t">path data objects</a>. +Each path data object is a <span class="stt">cairo_path_data_t</span>, +which is specified with a C union:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">union</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">_cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_type_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">type</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">length</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">header</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">point</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Helpfully, the FFI library comes with support for unions with the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span></span> type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It&rsquo;s pretty +straightforward to specify this type too:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the path data type is just an enum</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">move-to</span><span class="hspace">&nbsp;</span><span class="RktVal">line-to</span><span class="hspace">&nbsp;</span><span class="RktVal">curve-to</span><span class="hspace">&nbsp;</span><span class="RktVal">close-path</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the header case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the point case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>There&rsquo;s a new type constructor here so let me explain that first. +The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span></span> constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span>, this constructor doesn&rsquo;t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.</p> + +<p>Each of the path data structs in the path data array will be manipulated with +the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">create a union from a list of doubles</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1.3</span><span class="hspace">&nbsp;</span><span class="RktVal">5.8</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">source type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">target type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;union&gt;</span></p></td></tr></tbody></table></div> + +<p>This snippet first construct a union object (via the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> +type) using a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span></span>. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it&rsquo;s an easy way +to generate a union object.</p> + +<p>The second line shows that a union prints as an opaque object. You can&rsquo;t do +anything with a union in Racket unless you project it to one of the sub-cases with +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></span> function. +This projection is <span class="emph">unsafe</span>, in the sense that if you don&rsquo;t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.</p> + +<p>More concretely, let&rsquo;s see what happens if we try to extract a value out of the +union both correctly and incorrectly:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">correct (matches construction)</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">cases are zero-indexed and ordered as written</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(1.3 5.8)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">incorrect, error</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">enum:int-&gt;_cairo_path_data_type_t: expected a known</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">#&lt;ctype:ufixint&gt;, got: 3435973837</span></p></td></tr></tbody></table></div> + +<p>Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.</p> + +<p>With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.</p> + +<p>With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it&rsquo;s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.</p> + +<p>So before moving on, let&rsquo;s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we&rsquo;ll figure out how to +deal with the array itself.</p> + +<h1><a name="(part._.Some_low-level_operations)"></a>Some low-level operations</h1> + +<p>Since we still don&rsquo;t have a C type for <span class="stt">cairo_path_t</span>, let&rsquo;s go ahead +and make a simple one where we punt on the work of specifying the array +type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In this type, we have specified the array as a bare <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span>. +For some added safety, we could also use something like +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__cpointer%29%29">_cpointer</a></span><span class="stt"> </span><span class="RktVal">'</span><span class="RktVal">cairo_status_t</span><span class="RktPn">)</span></span>, which sets up a tagged pointer +type like we saw in the first blog post with +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>.</p> + +<p>We&rsquo;ve seen the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type before, but haven&rsquo;t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.</p> + +<p>Before we get to that, let&rsquo;s go ahead and set up an FFI binding for +<span class="stt">cairo_copy_path</span> so that we can obtain a path struct to manipulate:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Do stuff to make the current</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">path non-empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Get the current path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/set_.html#%28form._%28%28quote._~23~25kernel%29._set%21%29%29">set!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Stroke clears the path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">so do it last</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<p>Note that <span class="RktWrap"><span class="RktSym">cairo-copy-path</span></span> gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span>, which +lets you dereference a pointer and access it at some concrete C type.</p> + +<p><span style="font-weight: bold">Note:</span> the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> function also takes an optional +offset argument which we will be used in an example later.</p> + +<p>For example, we can use <span class="RktWrap"><span class="RktSym">a-path</span></span> as a <span class="RktWrap"><span class="RktSym">_simple_cairo_path_t</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(0 #&lt;cpointer&gt; 8)</span></p></td></tr></tbody></table></div> + +<p>And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> again with an array type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">array</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">length field</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">array</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(#&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt;)</span></p></td></tr></tbody></table></div> + +<p>The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(move-to 2)</span></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(50.0 50.0)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">nonsense data here, wrong union case</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(4.2439915824246e-314 0.0)</span></p></td></tr></tbody></table></div> + +<p>One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.</p> + +<p>An alternative is to define a <span class="emph">custom C type</span> that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don&rsquo;t need to think about applying helper functions and +dereferencing pointers.</p> + +<h1><a name="(part._.Custom_.C_types)"></a>Custom C types</h1> + +<p>I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.</p> + +<p>In this way, it&rsquo;s possible to conduct interesting conversions, such as +dereferencing union objects automatically.</p> + +<p>Now let&rsquo;s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.</p> + +<p>First, we&rsquo;ll start by defining a struct type for the Racket representation +of Cairo paths:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define-struct.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._struct%29%29">struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ptr</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:property</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._prop~3asequence%29%29">prop:sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The representation will store one field <span class="RktWrap"><span class="RktSym">ptr</span></span> which, as the name +suggests, will store a pointer value. We&rsquo;ll see what to do with this +pointer later.</p> + +<p>This definition uses a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/structprops.html#%28tech._structure._type._property%29"><span class="techinside">structure type property</span></a> to make instances of +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> automatically work as sequences. This means that you +can iterate over them with a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span></span> loop or apply <span class="RktWrap"><span class="RktSym">sequence-ref</span></span> +on them. The property takes a function that takes an instance of the struct +type itself (here <span class="RktWrap"><span class="RktSym">p</span></span>) and that returns a sequence.</p> + +<p>We&rsquo;ll later define the <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span> function that will actually +construct the relevant sequence for us. For now, let&rsquo;s see how to construct +the C type given this struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract pointer out of representation</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">racket-&gt;c</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Just apply the Racket constructor</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">c-&gt;racket</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">racket-&gt;c</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">c-&gt;racket</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The base type for this <span class="RktWrap"><span class="RktSym">_cairo_path_t</span></span> is a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type. Since +the Cairo API returns pointers to new path values, it&rsquo;s hard to avoid using some +kind of pointer type as the base type here.</p> + +<p>This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we&rsquo;ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.</p> + +<p>The real work is done by the helper function that makes a +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> instance work as a sequence.</p> + +<p>Starting top-down, let&rsquo;s look at the definition of <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Cairo-Path -&gt; Sequence</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match-define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/values.html#%28def._%28%28quote._~23~25kernel%29._values%29%29">values</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The first thing the function does is extract the pointer out of the +representation, and then immediately calls <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.</p> + +<p><span style="font-weight: bold">Note:</span> in case you&rsquo;re not very familiar with Racket pattern matching, the +<span class="RktWrap"><span class="RktSym">match-define</span></span> form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml&rsquo;s <span class="stt">let</span> statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +<a href="http://docs.racket-lang.org/reference/match.html#%28form._%28%28lib._racket%2Fmatch..rkt%29._match-define%29%29">docs</a>.</p> + +<p>After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> function. Essentially, <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.</p> + +<p><span style="font-weight: bold">Note:</span> technically <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it&rsquo;s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.</p> + +<p>In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the <span class="RktWrap"><span class="RktSym">pos-&gt;element</span></span> function +and its helpers:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer -&gt; Integer -&gt; Element</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract the data path header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Length includes header, so subtract 1</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._sub1%29%29">sub1</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._add1%29%29">add1</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer Integer Integer -&gt; (Listof Data)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">offset argument</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in <span class="RktWrap"><span class="RktSym">get-points</span></span> +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.</p> + +<p>Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Note that determining the end of the sequence is very easy. It&rsquo;s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="stt"> </span><span class="RktSym">pos</span><span class="stt"> </span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></span>.</p> + +<p>Now we can try using a path as a sequence:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Using path as a sequence</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">elem</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="hspace">&nbsp;</span><span class="RktSym">elem</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 50.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (206.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (115.0 115.0))</span></p></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span></span> type.</p> + +<p>That concludes part 3 of the FFI tutorial. Hopefully you&rsquo;re now equipped +to deal with union types and custom C types. If not, see the +<a href="http://docs.racket-lang.org/foreign/index.html">FFI reference</a> +for more details on +<a href="http://docs.racket-lang.org/foreign/C_Union_Types.html">unions</a> +and +<a href="http://docs.racket-lang.org/foreign/ctype.html">custom C types</a>.</p> + +<p><span class="emph">Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!</span></p> + + Tutorial: Racket FFI, Part 2 + + urn:http-prl-ccs-neu-edu:-blog-2016-06-29-tutorial-racket-ffi-part-2 + 2016-06-29T18:48:17Z + 2016-06-29T18:48:17Z + + Asumu Takikawa + +<p>This is part 2 of my tutorial on using the Racket FFI. If you haven&rsquo;t read +part 1 yet, you can find it +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a>. +<span style="font-weight: bold">Update:</span> part 3 is also now available +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.</p> +<!--more--> + +<p>First, here&rsquo;s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">bitmap magic</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_create</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Bitmap -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a helper for displaying the bitmap</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<h1><a name="(part._.Dashes_and_array_arguments)"></a>Dashes and array arguments</h1> + +<p>To start off, let&rsquo;s look at another C example from the Cairo +<a href="https://www.cairographics.org/samples/">samples page</a>. +This time we will look at the "dash" example, which has a +use of an input array:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">sizeof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">)</span><span class="RktMeta">/sizeof</span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktVal">0</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">50.0</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">25.6</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_rel_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">102.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_curve_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The most interesting function here is <span class="stt">cairo_set_dash</span>, which takes an +array argument. The only other new functions are <span class="stt">cairo_rel_line_to</span> +and <span class="stt">cairo_curve_to</span> which have very straightforward C types:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-rel-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_rel_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-curve-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_curve_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Meanwhile, the C type signature for <span class="stt">cairo_set_dash</span> from the Cairo +docs looks like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Something to note about the arguments is that <span class="stt">num_dashes</span> +encodes the length of the array <span class="stt">dashes</span>. This will come up later when +we want to make the C type for this function more convenient.</p> + +<p>On the Racket side, it&rsquo;s natural to represent the array of dashes as either a +list or <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/vectors.html#%28tech._vector%29"><span class="techinside">vector</span></a> of numbers. Given that, a fairly literal translation of +the C type above might look like the following:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This type includes a type constructor we haven&rsquo;t seen yet: <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span>. +This is a so-called <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28tech._custom._function._type%29"><span class="techinside">custom function type</span></a> that has special meaning +inside of a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.</p> + +<p>Since we only want to provide a list from the Racket side to C, we&rsquo;ll use +the <span class="RktWrap"><span class="RktSym">i</span></span> input mode. We can then call the function like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">4</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Note that because of how we defined the type of <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it&rsquo;s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.</p> + +<p>Luckily, the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type constructor actually lets you do this with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">name</span><span class="stt"> </span><span class="RktSym">:</span><span class="stt"> </span><span class="RktSym">type</span><span class="RktPn">)</span></span> syntax for naming arguments in combination with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">type</span><span class="stt"> </span><span class="RktVar">=</span><span class="stt"> </span><span class="RktSym">expr</span><span class="RktPn">)</span></span> syntax for supplying computed arguments:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">name this argument for later uses in the type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a computed argument position</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3d%29%29">=</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>When a computed argument is specified with a <span class="RktWrap"><span class="RktVar">=</span></span>, it&rsquo;s not necessary to provide +the argument on the Racket side. So <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> is now an arity 3 +function that can be called like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This means we&rsquo;ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it&rsquo;s also possible to use Racket vectors instead of lists by using +the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__vector%29%29">_vector</a></span></span> type constructor.</p> + +<p>Putting it all together, we can reproduce the dashes example like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">25.6</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-rel-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-1</span>02.4</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-curve-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Result_arguments_and_.C_structs)"></a>Result arguments and C structs</h1> + +<p>For some more advanced FFI hacking, let&rsquo;s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Our challenge is to make a Racket function that takes a string (let&rsquo;s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let&rsquo;s just scale the +text to fit the width and assume the height will be okay.</p> + +<p>To implement the key step of measuring the text size, we can use the +<a href="https://www.cairographics.org/manual/cairo-text.html#cairo-text-extents"><span class="stt">cairo_text_extents</span></a> +function. Its type signature is as follows:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">char</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*utf8,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*extents</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The interesting part of this signature is that +<a href="https://www.cairographics.org/manual/cairo-cairo-scaled-font-t.html#cairo-text-extents-t"><span class="stt">cairo_text_extents_t</span></a> +is a struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">Cairo</span><span class="hspace">&nbsp;</span><span class="RktCmt">docs</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">height</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>We haven&rsquo;t yet seen how to handle C structs with the FFI, but it&rsquo;s not +too tricky. Support for C structs comes built-in and will look familiar +if you&rsquo;re used to Racket structs. We can directly translate the documented +definition above into a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span> declaration:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the leading underscore is mandatory</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">width</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">height</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">x-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">pointer to struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">allows NULL pointer</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer/null</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr></tbody></table></div> + +<p>Along with functions that look like regular Racket struct operations:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a struct constructor</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:make-cairo_text_extents_t&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field selector</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:cairo_text_extents_t-width&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a predicate for the struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t?</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:^TYPE?&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field mutation function</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">set-cairo_text_extents_t-width!</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:set-cairo_text_extents_t-width!&gt;</span></p></td></tr></tbody></table></div> + +<p>With the struct type defined, it&rsquo;s easy to come up with a rudimentary +interface for <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>This style of programming feels awfully imperative though. Since we&rsquo;re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span> FFI wrapper +by combining named arguments, a new <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> type constructor, +and a neat feature of <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> that lets you customize +the return result:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">named args and _ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">ext</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the return result of the C function</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">custom return result for the wrapper</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">ext</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> constructor works like the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span> constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the <span class="RktWrap"><span class="RktSym">o</span></span> mode to <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span>. +In output mode, this type will automatically allocate a new instance of the type +(using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._malloc%29%29">malloc</a></span></span> function) and arrange for it to be passed in as +a pointer.</p> + +<p>The strangest part of this example is that there are now two uses of the +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span></span> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.</p> + +<p>Using this new version of the wrapper is much simpler:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>With that in hand, it&rsquo;s pretty easy to write the function we set out to write:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-show-text</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_show_text</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-scale</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">String -&gt; Void</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">draws a string scaled horizontally</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">20</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">2.0</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._-%29%29"><span class="nobreak">-</span></a></span><span class="hspace">&nbsp;</span><span class="RktVal">256.0</span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym">width</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-scale</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-show-text</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>And to conclude part 2 of this tutorial, here&rsquo;s an example use +of the new <span class="RktWrap"><span class="RktSym">fit-text</span></span> function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktVal">"Saluton, Mondo / Hallo, mundo"</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + + Tutorial: Using Racket's FFI + + urn:http-prl-ccs-neu-edu:-blog-2016-06-27-tutorial-using-racket-s-ffi + 2016-06-27T16:22:11Z + 2016-06-27T16:22:11Z + + Asumu Takikawa + +<p><span style="font-weight: bold">Update:</span> this post is now part of a series. Part 2 is +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a> +and part 3 is +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>I&rsquo;ve seen several people ask for a tutorial on Racket&rsquo;s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +<a href="http://docs.racket-lang.org/foreign/index.html">documentation</a> +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.</p> + +<p>With that in mind, this blog post will provide a step-by-step tutorial +for Racket&rsquo;s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.</p> +<!--more--> + +<p>Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop <a href="http://www.ccs.neu.edu/racket/pubs/scheme04-bo.pdf">paper</a> +that you can read if you&rsquo;re curious about the design.</p> + +<p>The tutorial will focus on using the <a href="https://www.cairographics.org/">Cairo</a> +graphics library, mainly because it comes bundled with Racket.</p> + +<p>To start, let&rsquo;s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo&rsquo;s +<a href="https://www.cairographics.org/samples/">samples page</a>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">30.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">CAIRO_LINE_CAP_ROUND</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This uses the Racket drawing library <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/draw/index.html"><span class="RktSym">racket/draw</span></a></span> to construct +a bitmap object that we&rsquo;ll draw on. The <span class="RktWrap"><span class="RktSym">get-handle</span></span> method just extracts a +low-level Cairo surface value that we can use.</p> + +<p>NB: these code snippets don&rsquo;t come with a <span class="stt">#lang</span> declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy &amp; paste the snippets into your REPL.</p> + +<p>Our first real step is to import the FFI itself:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>As the module name suggests, the FFI is <span class="emph">unsafe</span> and can cause your Racket process +to segfault. If you&rsquo;re following along in DrRacket, you will want to save your file +frequently.</p> + +<p>Next, we can load the Cairo library to obtain a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28tech._foreign._library._value%29"><span class="techinside">foreign-library value</span></a>, which +is a handle that we use to access C values and functions:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Since Cairo has already been loaded by the Racket process because of the +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/gui/index.html"><span class="RktSym">racket/gui</span></a></span> import earlier, we can supply <span class="RktWrap"><span class="RktVal">#f</span></span> here as an +argument to <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span>. Normally you supply the name of a shared +library file such as <span class="RktWrap"><span class="RktVal">"libcairo"</span></span>:</p> + +<div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"libcairo"</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"2"</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></p></div> + +<p>The last list argument specifies the accepted versions (<span class="RktWrap"><span class="RktVal">#f</span></span> allows +a version-less library). For this post, those details aren&rsquo;t important but +see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span> if you&rsquo;re curious.</p> + +<h1><a name="(part._.Extracting_functions)"></a>Extracting functions</h1> + +<p>Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> function. The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> +function takes three arguments:</p> + +<ul> + <li> + <p>The name of the value as a string (or symbol or bytestring)</p></li> + <li> + <p>a foreign library value, and</p></li> + <li> + <p>a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/types.html#%28tech._c._type%29"><span class="techinside">C type</span></a>, which is a type description that tells +the FFI how to marshall between Racket and C.</p></li></ul> + +<p>C types are a crucial concept for the FFI. They range from relatively +simple types like <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> to more complicated +type constructors such as <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span>. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span></span> with two functions that +handle marshalling between C and Racket code.</p> + +<p>To make progress with our Cairo code, we need to create a drawing context from +the surface object <span class="RktWrap"><span class="RktSym">bt-surface</span></span> that we defined a while ago. The +relevant function in the Cairo docs is +<a href="https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-create"><span class="stt">cairo_create</span></a>, +which has the following type signature:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">NB:</span><span class="hspace">&nbsp;</span><span class="RktCmt">this</span><span class="hspace">&nbsp;</span><span class="RktCmt">is</span><span class="hspace">&nbsp;</span><span class="RktCmt">C</span><span class="hspace">&nbsp;</span><span class="RktCmt">code</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_create</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_surface_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*target</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a <span class="stt">cairo_surface_t</span> and +returns a pointer to a <span class="stt">cairo_t</span>. Let&rsquo;s start with a very simple C type +that matches up with this behavior: </div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span></p></div></div> + +<div class="SIntrapara">This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type.</div> + +<p>The following definition shows how to use this type to obtain a foreign +function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Then we can use <span class="RktWrap"><span class="RktSym">cairo-create</span></span> as an ordinary racket function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">ctx</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Interlude__more_type_safety)"></a>Interlude: more type safety</h1> + +<p>Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> types, it is easy +to accidentally misuse the function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">You may not want to actually run this</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a cairo_t is not a cairo_surface_t</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>To prevent such bad uses, it is good practice to use <span class="emph">tagged</span> pointer types +using <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>. Here are two example definitions that +correspond to the <span class="stt">cairo_t</span> and <span class="stt">cairo_surface_t</span> types from earlier:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">The leading underscores are mandatory</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>We can then redefine <span class="RktWrap"><span class="RktSym">cairo-create</span></span> with a better type, which will +prevent ill-typed calls:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">cairo_surface_t-&gt;C: argument is not non-null</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">`cairo_surface_t' pointer</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktErr">argument: #&lt;cpointer:cairo_t&gt;</span></p></td></tr></tbody></table></div> + +<p>Unfortunately our old definition of <span class="RktWrap"><span class="RktSym">ctx</span></span> doesn&rsquo;t have this tag:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#f</span></p></td></tr></tbody></table></div> + +<p>Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it&rsquo;s also possible to update +existing pointers with a tag like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-push-tag%21%29%29">cpointer-push-tag!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#t</span></p></td></tr></tbody></table></div> + +<p>Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).</p> + +<h1><a name="(part._.Macros_for_reducing_boilerplate)"></a>Macros for reducing boilerplate</h1> + +<p>Now let&rsquo;s start building the FFI bindings for the functions in the Cairo sample. +First, let&rsquo;s go ahead and look at all of the types for the sample functions +from the C API docs:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_cap_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">line_cap</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Starting with <span class="stt">cairo_move_to</span>, we can set up a definition like we did +with <span class="stt">cairo_create</span> before:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">"cairo_move_to"</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span> library that help reduce the +verbosity. Here&rsquo;s an alternative definition of <span class="RktWrap"><span class="RktSym">cairo-move-to</span></span> +using the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form from +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>As you can see, the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +<span class="RktWrap"><span class="RktSym">cairo_move_to</span></span>) you also don&rsquo;t need to supply the C name either.</p> + +<p>The definitions for <span class="stt">cairo_line_to</span>, <span class="stt">cairo_set_line_width</span>, and +<span class="stt">cairo_stroke</span> aren&rsquo;t very interesting, so I&rsquo;ll just include them +below without comment:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="stt">cairo_set_line_cap</span> case is more interesting because the type +<span class="stt">cairo_line_cap_t</span> is a C enumeration type. Racket&rsquo;s FFI comes with +convenience forms for defining enumeration types&#8212; + <wbr />though it&rsquo;s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here&rsquo;s a quote from the Barzilay and Orlovsky +paper on that:</p> + +<blockquote class="SubFlow"> + <p>Our design follows a simple principle: keep C-level +functionality to a minimum.</p></blockquote> + +<p>and specifically about enumerations:</p> + +<blockquote class="SubFlow"> + <p>For example, the C level part of our interface does not commit to a +specific implementation for enumerations &#8212; it simply exposes C integers.</p></blockquote> + +<p>To define an enumeration, we can use the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the <span class="stt">cairo_line_cap_t</span> type, it suffices to +just supply the cases as a list of symbols:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It&rsquo;s also possible to specify how the symbols map to integers more precisely +(see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> for those details).</p> + +<p>Given this type, we can specify the type for the line cap function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<h1><a name="(part._.Putting_it_all_together)"></a>Putting it all together</h1> + +<p>Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">30.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-cap</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">round</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object <span class="RktWrap"><span class="RktSym">bt</span></span>. To see the +results of what we drew, we can just evaluate <span class="RktWrap"><span class="RktSym">bt</span></span> at the REPL. But it&rsquo;s +a little nicer if we use the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> library to draw a frame around +it to distinguish it from the background:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>And we&rsquo;re done! Of course, there is a lot more to the FFI. For example, I haven&rsquo;t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I&rsquo;m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!</p> \ No newline at end of file diff --git a/blog/feeds/FFI.rss.xml b/blog/feeds/FFI.rss.xml new file mode 100644 index 00000000..cde62c8a --- /dev/null +++ b/blog/feeds/FFI.rss.xml @@ -0,0 +1,2199 @@ + + + + PRL Blog: Posts tagged 'FFI' + PRL Blog: Posts tagged 'FFI' + http://prl.ccs.neu.edu/blog/tags/FFI.html + Mon, 11 Jul 2016 17:33:40 UT + Mon, 11 Jul 2016 17:33:40 UT + 1800 + + Tutorial: Racket FFI, part 3 + http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/?utm_source=FFI&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-07-11-tutorial-racket-ffi-part-3 + Mon, 11 Jul 2016 17:33:40 UT + Asumu Takikawa + +<p>This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a> +and part 2 +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a>.</p> + +<p>In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.</p> +<!--more--> + +<p>As in the second post, let&rsquo;s start with some prologue code that establishes +the definitions from the previous two posts. But first, I&rsquo;m getting tired of +writing the <span class="stt">#:c-id identifier</span> notation for the underscored C function +names.</p> + +<p>Instead, let&rsquo;s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">$</span><span class="hspace">&nbsp;</span><span class="RktMeta">raco</span><span class="hspace">&nbsp;</span><span class="RktMeta">pkg</span><span class="hspace">&nbsp;</span><span class="RktMeta">install</span><span class="hspace">&nbsp;</span><span class="RktMeta">ffi-definer-convention</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>or you can just execute the following snippet in Racket:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pkg</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">pkg-install-command</span><span class="hspace">&nbsp;</span><span class="RktPn">#:skip-installed</span><span class="hspace">&nbsp;</span><span class="RktVal">#t</span><span class="hspace">&nbsp;</span><span class="RktVal">"ffi-definer-convention"</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Downloading repository git://github.com/takikawa/racket-ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: version: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: platform: x86_64-linux [3m]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: target machine: racket</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: installation name: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: variants: 3m</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main collects: /usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: collects paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main pkgs: /usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: pkgs paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: links files: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main docs: /usr/share/racket/doc</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- updating info-domain tables ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- pre-installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing foreign libraries --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing shared files ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- compiling collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- parallel build using 4 jobs ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 making: &lt;pkgs&gt;/ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- creating launchers --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing man pages --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- building documentation --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 running: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 rendering: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 2 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/local-redirect.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 1 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/release.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/search.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/start.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- post-installing collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This will install the package and compile its contents. If you&rsquo;re curious, the +docs for the package are available +<a href="http://docs.racket-lang.org/ffi-definer-convention/index.html">here</a>.</p> + +<p><span style="font-weight: bold">Note:</span> if you&rsquo;ve never installed a package before, you may want to glance +at the +<a href="http://docs.racket-lang.org/pkg/getting-started.html">package system docs</a>. +A tl;dr of packages is that they bundle Racket <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/collects.html#%28tech._collection%29"><span class="techinside">collections</span></a>, which are +sets of modules that you can refer to in a location-independent fashion such as +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> or <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28mod-path._racket%2Flist%29"><span class="RktSym">racket/list</span></a></span>.</p> + +<p>Anyhow, here&rsquo;s the prologue code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">avoid conflict with below</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._except-in%29%29">except-in</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">define-ffi-definer</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the new 3rd-party pkg</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi-definer-convention</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">describes how to transform from</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Racket to C ids</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:make-c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">convention:hyphen-&gt;underscore</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">note lack of #:c-id keyword arguments</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">(_cairo_t -&gt; Void) -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">do some drawing and give us the pict</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">f</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">f</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Notice that the <span class="RktWrap"><span class="RktSym">define-cairo</span></span> forms don&rsquo;t have any <span class="stt">#:c-id</span> keywords +anymore. Instead, the prologue code uses an overriden <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> +from my package that supports a <span class="stt">#:make-c-id</span> keyword that lets you specify +a naming convention to follow.</p> + +<p>Also, instead of creating a single bitmap and drawing into it, we now have a +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> function that takes a drawing function. When called, +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> will call the given function with a new bitmap object and +return the result.</p> + +<p>Now let&rsquo;s get to the main point of this blog post. Let&rsquo;s say that we want to play +with Cairo <a href="https://www.cairographics.org/manual/cairo-Paths.html">path</a> +objects this time. A path is +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-t">defined</a> +as a struct with the following structure:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_status_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">status</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it&rsquo;s +useful to define C types for the types of values in the path struct&rsquo;s fields. First, +let&rsquo;s specify that a <span class="stt">cairo_status_t</span> is an integer type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>It&rsquo;s actually an enum, but for the examples in this post we don&rsquo;t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-data-t">path data objects</a>. +Each path data object is a <span class="stt">cairo_path_data_t</span>, +which is specified with a C union:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">union</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">_cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_type_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">type</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">length</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">header</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">point</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Helpfully, the FFI library comes with support for unions with the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span></span> type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It&rsquo;s pretty +straightforward to specify this type too:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the path data type is just an enum</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">move-to</span><span class="hspace">&nbsp;</span><span class="RktVal">line-to</span><span class="hspace">&nbsp;</span><span class="RktVal">curve-to</span><span class="hspace">&nbsp;</span><span class="RktVal">close-path</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the header case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the point case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>There&rsquo;s a new type constructor here so let me explain that first. +The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span></span> constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span>, this constructor doesn&rsquo;t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.</p> + +<p>Each of the path data structs in the path data array will be manipulated with +the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">create a union from a list of doubles</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1.3</span><span class="hspace">&nbsp;</span><span class="RktVal">5.8</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">source type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">target type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;union&gt;</span></p></td></tr></tbody></table></div> + +<p>This snippet first construct a union object (via the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> +type) using a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span></span>. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it&rsquo;s an easy way +to generate a union object.</p> + +<p>The second line shows that a union prints as an opaque object. You can&rsquo;t do +anything with a union in Racket unless you project it to one of the sub-cases with +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></span> function. +This projection is <span class="emph">unsafe</span>, in the sense that if you don&rsquo;t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.</p> + +<p>More concretely, let&rsquo;s see what happens if we try to extract a value out of the +union both correctly and incorrectly:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">correct (matches construction)</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">cases are zero-indexed and ordered as written</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(1.3 5.8)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">incorrect, error</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">enum:int-&gt;_cairo_path_data_type_t: expected a known</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">#&lt;ctype:ufixint&gt;, got: 3435973837</span></p></td></tr></tbody></table></div> + +<p>Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.</p> + +<p>With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.</p> + +<p>With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it&rsquo;s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.</p> + +<p>So before moving on, let&rsquo;s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we&rsquo;ll figure out how to +deal with the array itself.</p> + +<h1><a name="(part._.Some_low-level_operations)"></a>Some low-level operations</h1> + +<p>Since we still don&rsquo;t have a C type for <span class="stt">cairo_path_t</span>, let&rsquo;s go ahead +and make a simple one where we punt on the work of specifying the array +type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In this type, we have specified the array as a bare <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span>. +For some added safety, we could also use something like +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__cpointer%29%29">_cpointer</a></span><span class="stt"> </span><span class="RktVal">'</span><span class="RktVal">cairo_status_t</span><span class="RktPn">)</span></span>, which sets up a tagged pointer +type like we saw in the first blog post with +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>.</p> + +<p>We&rsquo;ve seen the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type before, but haven&rsquo;t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.</p> + +<p>Before we get to that, let&rsquo;s go ahead and set up an FFI binding for +<span class="stt">cairo_copy_path</span> so that we can obtain a path struct to manipulate:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Do stuff to make the current</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">path non-empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Get the current path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/set_.html#%28form._%28%28quote._~23~25kernel%29._set%21%29%29">set!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Stroke clears the path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">so do it last</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<p>Note that <span class="RktWrap"><span class="RktSym">cairo-copy-path</span></span> gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span>, which +lets you dereference a pointer and access it at some concrete C type.</p> + +<p><span style="font-weight: bold">Note:</span> the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> function also takes an optional +offset argument which we will be used in an example later.</p> + +<p>For example, we can use <span class="RktWrap"><span class="RktSym">a-path</span></span> as a <span class="RktWrap"><span class="RktSym">_simple_cairo_path_t</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(0 #&lt;cpointer&gt; 8)</span></p></td></tr></tbody></table></div> + +<p>And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> again with an array type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">array</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">length field</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">array</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(#&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt;)</span></p></td></tr></tbody></table></div> + +<p>The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(move-to 2)</span></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(50.0 50.0)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">nonsense data here, wrong union case</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(4.2439915824246e-314 0.0)</span></p></td></tr></tbody></table></div> + +<p>One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.</p> + +<p>An alternative is to define a <span class="emph">custom C type</span> that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don&rsquo;t need to think about applying helper functions and +dereferencing pointers.</p> + +<h1><a name="(part._.Custom_.C_types)"></a>Custom C types</h1> + +<p>I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.</p> + +<p>In this way, it&rsquo;s possible to conduct interesting conversions, such as +dereferencing union objects automatically.</p> + +<p>Now let&rsquo;s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.</p> + +<p>First, we&rsquo;ll start by defining a struct type for the Racket representation +of Cairo paths:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define-struct.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._struct%29%29">struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ptr</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:property</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._prop~3asequence%29%29">prop:sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The representation will store one field <span class="RktWrap"><span class="RktSym">ptr</span></span> which, as the name +suggests, will store a pointer value. We&rsquo;ll see what to do with this +pointer later.</p> + +<p>This definition uses a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/structprops.html#%28tech._structure._type._property%29"><span class="techinside">structure type property</span></a> to make instances of +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> automatically work as sequences. This means that you +can iterate over them with a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span></span> loop or apply <span class="RktWrap"><span class="RktSym">sequence-ref</span></span> +on them. The property takes a function that takes an instance of the struct +type itself (here <span class="RktWrap"><span class="RktSym">p</span></span>) and that returns a sequence.</p> + +<p>We&rsquo;ll later define the <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span> function that will actually +construct the relevant sequence for us. For now, let&rsquo;s see how to construct +the C type given this struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract pointer out of representation</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">racket-&gt;c</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Just apply the Racket constructor</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">c-&gt;racket</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">racket-&gt;c</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">c-&gt;racket</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The base type for this <span class="RktWrap"><span class="RktSym">_cairo_path_t</span></span> is a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type. Since +the Cairo API returns pointers to new path values, it&rsquo;s hard to avoid using some +kind of pointer type as the base type here.</p> + +<p>This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we&rsquo;ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.</p> + +<p>The real work is done by the helper function that makes a +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> instance work as a sequence.</p> + +<p>Starting top-down, let&rsquo;s look at the definition of <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Cairo-Path -&gt; Sequence</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match-define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/values.html#%28def._%28%28quote._~23~25kernel%29._values%29%29">values</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The first thing the function does is extract the pointer out of the +representation, and then immediately calls <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.</p> + +<p><span style="font-weight: bold">Note:</span> in case you&rsquo;re not very familiar with Racket pattern matching, the +<span class="RktWrap"><span class="RktSym">match-define</span></span> form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml&rsquo;s <span class="stt">let</span> statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +<a href="http://docs.racket-lang.org/reference/match.html#%28form._%28%28lib._racket%2Fmatch..rkt%29._match-define%29%29">docs</a>.</p> + +<p>After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> function. Essentially, <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.</p> + +<p><span style="font-weight: bold">Note:</span> technically <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it&rsquo;s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.</p> + +<p>In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the <span class="RktWrap"><span class="RktSym">pos-&gt;element</span></span> function +and its helpers:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer -&gt; Integer -&gt; Element</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract the data path header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Length includes header, so subtract 1</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._sub1%29%29">sub1</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._add1%29%29">add1</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer Integer Integer -&gt; (Listof Data)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">offset argument</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in <span class="RktWrap"><span class="RktSym">get-points</span></span> +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.</p> + +<p>Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Note that determining the end of the sequence is very easy. It&rsquo;s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="stt"> </span><span class="RktSym">pos</span><span class="stt"> </span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></span>.</p> + +<p>Now we can try using a path as a sequence:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Using path as a sequence</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">elem</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="hspace">&nbsp;</span><span class="RktSym">elem</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 50.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (206.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (115.0 115.0))</span></p></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span></span> type.</p> + +<p>That concludes part 3 of the FFI tutorial. Hopefully you&rsquo;re now equipped +to deal with union types and custom C types. If not, see the +<a href="http://docs.racket-lang.org/foreign/index.html">FFI reference</a> +for more details on +<a href="http://docs.racket-lang.org/foreign/C_Union_Types.html">unions</a> +and +<a href="http://docs.racket-lang.org/foreign/ctype.html">custom C types</a>.</p> + +<p><span class="emph">Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!</span></p> + + Tutorial: Racket FFI, Part 2 + http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/?utm_source=FFI&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-29-tutorial-racket-ffi-part-2 + Wed, 29 Jun 2016 18:48:17 UT + Asumu Takikawa + +<p>This is part 2 of my tutorial on using the Racket FFI. If you haven&rsquo;t read +part 1 yet, you can find it +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a>. +<span style="font-weight: bold">Update:</span> part 3 is also now available +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.</p> +<!--more--> + +<p>First, here&rsquo;s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">bitmap magic</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_create</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Bitmap -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a helper for displaying the bitmap</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<h1><a name="(part._.Dashes_and_array_arguments)"></a>Dashes and array arguments</h1> + +<p>To start off, let&rsquo;s look at another C example from the Cairo +<a href="https://www.cairographics.org/samples/">samples page</a>. +This time we will look at the "dash" example, which has a +use of an input array:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">sizeof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">)</span><span class="RktMeta">/sizeof</span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktVal">0</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">50.0</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">25.6</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_rel_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">102.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_curve_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The most interesting function here is <span class="stt">cairo_set_dash</span>, which takes an +array argument. The only other new functions are <span class="stt">cairo_rel_line_to</span> +and <span class="stt">cairo_curve_to</span> which have very straightforward C types:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-rel-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_rel_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-curve-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_curve_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Meanwhile, the C type signature for <span class="stt">cairo_set_dash</span> from the Cairo +docs looks like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Something to note about the arguments is that <span class="stt">num_dashes</span> +encodes the length of the array <span class="stt">dashes</span>. This will come up later when +we want to make the C type for this function more convenient.</p> + +<p>On the Racket side, it&rsquo;s natural to represent the array of dashes as either a +list or <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/vectors.html#%28tech._vector%29"><span class="techinside">vector</span></a> of numbers. Given that, a fairly literal translation of +the C type above might look like the following:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This type includes a type constructor we haven&rsquo;t seen yet: <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span>. +This is a so-called <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28tech._custom._function._type%29"><span class="techinside">custom function type</span></a> that has special meaning +inside of a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.</p> + +<p>Since we only want to provide a list from the Racket side to C, we&rsquo;ll use +the <span class="RktWrap"><span class="RktSym">i</span></span> input mode. We can then call the function like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">4</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Note that because of how we defined the type of <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it&rsquo;s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.</p> + +<p>Luckily, the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type constructor actually lets you do this with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">name</span><span class="stt"> </span><span class="RktSym">:</span><span class="stt"> </span><span class="RktSym">type</span><span class="RktPn">)</span></span> syntax for naming arguments in combination with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">type</span><span class="stt"> </span><span class="RktVar">=</span><span class="stt"> </span><span class="RktSym">expr</span><span class="RktPn">)</span></span> syntax for supplying computed arguments:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">name this argument for later uses in the type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a computed argument position</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3d%29%29">=</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>When a computed argument is specified with a <span class="RktWrap"><span class="RktVar">=</span></span>, it&rsquo;s not necessary to provide +the argument on the Racket side. So <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> is now an arity 3 +function that can be called like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This means we&rsquo;ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it&rsquo;s also possible to use Racket vectors instead of lists by using +the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__vector%29%29">_vector</a></span></span> type constructor.</p> + +<p>Putting it all together, we can reproduce the dashes example like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">25.6</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-rel-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-1</span>02.4</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-curve-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Result_arguments_and_.C_structs)"></a>Result arguments and C structs</h1> + +<p>For some more advanced FFI hacking, let&rsquo;s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Our challenge is to make a Racket function that takes a string (let&rsquo;s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let&rsquo;s just scale the +text to fit the width and assume the height will be okay.</p> + +<p>To implement the key step of measuring the text size, we can use the +<a href="https://www.cairographics.org/manual/cairo-text.html#cairo-text-extents"><span class="stt">cairo_text_extents</span></a> +function. Its type signature is as follows:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">char</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*utf8,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*extents</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The interesting part of this signature is that +<a href="https://www.cairographics.org/manual/cairo-cairo-scaled-font-t.html#cairo-text-extents-t"><span class="stt">cairo_text_extents_t</span></a> +is a struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">Cairo</span><span class="hspace">&nbsp;</span><span class="RktCmt">docs</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">height</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>We haven&rsquo;t yet seen how to handle C structs with the FFI, but it&rsquo;s not +too tricky. Support for C structs comes built-in and will look familiar +if you&rsquo;re used to Racket structs. We can directly translate the documented +definition above into a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span> declaration:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the leading underscore is mandatory</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">width</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">height</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">x-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">pointer to struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">allows NULL pointer</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer/null</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr></tbody></table></div> + +<p>Along with functions that look like regular Racket struct operations:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a struct constructor</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:make-cairo_text_extents_t&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field selector</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:cairo_text_extents_t-width&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a predicate for the struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t?</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:^TYPE?&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field mutation function</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">set-cairo_text_extents_t-width!</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:set-cairo_text_extents_t-width!&gt;</span></p></td></tr></tbody></table></div> + +<p>With the struct type defined, it&rsquo;s easy to come up with a rudimentary +interface for <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>This style of programming feels awfully imperative though. Since we&rsquo;re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span> FFI wrapper +by combining named arguments, a new <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> type constructor, +and a neat feature of <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> that lets you customize +the return result:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">named args and _ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">ext</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the return result of the C function</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">custom return result for the wrapper</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">ext</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> constructor works like the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span> constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the <span class="RktWrap"><span class="RktSym">o</span></span> mode to <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span>. +In output mode, this type will automatically allocate a new instance of the type +(using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._malloc%29%29">malloc</a></span></span> function) and arrange for it to be passed in as +a pointer.</p> + +<p>The strangest part of this example is that there are now two uses of the +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span></span> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.</p> + +<p>Using this new version of the wrapper is much simpler:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>With that in hand, it&rsquo;s pretty easy to write the function we set out to write:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-show-text</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_show_text</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-scale</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">String -&gt; Void</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">draws a string scaled horizontally</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">20</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">2.0</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._-%29%29"><span class="nobreak">-</span></a></span><span class="hspace">&nbsp;</span><span class="RktVal">256.0</span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym">width</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-scale</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-show-text</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>And to conclude part 2 of this tutorial, here&rsquo;s an example use +of the new <span class="RktWrap"><span class="RktSym">fit-text</span></span> function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktVal">"Saluton, Mondo / Hallo, mundo"</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + + Tutorial: Using Racket's FFI + http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/?utm_source=FFI&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-27-tutorial-using-racket-s-ffi + Mon, 27 Jun 2016 16:22:11 UT + Asumu Takikawa + +<p><span style="font-weight: bold">Update:</span> this post is now part of a series. Part 2 is +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a> +and part 3 is +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>I&rsquo;ve seen several people ask for a tutorial on Racket&rsquo;s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +<a href="http://docs.racket-lang.org/foreign/index.html">documentation</a> +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.</p> + +<p>With that in mind, this blog post will provide a step-by-step tutorial +for Racket&rsquo;s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.</p> +<!--more--> + +<p>Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop <a href="http://www.ccs.neu.edu/racket/pubs/scheme04-bo.pdf">paper</a> +that you can read if you&rsquo;re curious about the design.</p> + +<p>The tutorial will focus on using the <a href="https://www.cairographics.org/">Cairo</a> +graphics library, mainly because it comes bundled with Racket.</p> + +<p>To start, let&rsquo;s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo&rsquo;s +<a href="https://www.cairographics.org/samples/">samples page</a>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">30.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">CAIRO_LINE_CAP_ROUND</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This uses the Racket drawing library <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/draw/index.html"><span class="RktSym">racket/draw</span></a></span> to construct +a bitmap object that we&rsquo;ll draw on. The <span class="RktWrap"><span class="RktSym">get-handle</span></span> method just extracts a +low-level Cairo surface value that we can use.</p> + +<p>NB: these code snippets don&rsquo;t come with a <span class="stt">#lang</span> declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy &amp; paste the snippets into your REPL.</p> + +<p>Our first real step is to import the FFI itself:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>As the module name suggests, the FFI is <span class="emph">unsafe</span> and can cause your Racket process +to segfault. If you&rsquo;re following along in DrRacket, you will want to save your file +frequently.</p> + +<p>Next, we can load the Cairo library to obtain a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28tech._foreign._library._value%29"><span class="techinside">foreign-library value</span></a>, which +is a handle that we use to access C values and functions:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Since Cairo has already been loaded by the Racket process because of the +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/gui/index.html"><span class="RktSym">racket/gui</span></a></span> import earlier, we can supply <span class="RktWrap"><span class="RktVal">#f</span></span> here as an +argument to <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span>. Normally you supply the name of a shared +library file such as <span class="RktWrap"><span class="RktVal">"libcairo"</span></span>:</p> + +<div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"libcairo"</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"2"</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></p></div> + +<p>The last list argument specifies the accepted versions (<span class="RktWrap"><span class="RktVal">#f</span></span> allows +a version-less library). For this post, those details aren&rsquo;t important but +see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span> if you&rsquo;re curious.</p> + +<h1><a name="(part._.Extracting_functions)"></a>Extracting functions</h1> + +<p>Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> function. The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> +function takes three arguments:</p> + +<ul> + <li> + <p>The name of the value as a string (or symbol or bytestring)</p></li> + <li> + <p>a foreign library value, and</p></li> + <li> + <p>a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/types.html#%28tech._c._type%29"><span class="techinside">C type</span></a>, which is a type description that tells +the FFI how to marshall between Racket and C.</p></li></ul> + +<p>C types are a crucial concept for the FFI. They range from relatively +simple types like <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> to more complicated +type constructors such as <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span>. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span></span> with two functions that +handle marshalling between C and Racket code.</p> + +<p>To make progress with our Cairo code, we need to create a drawing context from +the surface object <span class="RktWrap"><span class="RktSym">bt-surface</span></span> that we defined a while ago. The +relevant function in the Cairo docs is +<a href="https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-create"><span class="stt">cairo_create</span></a>, +which has the following type signature:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">NB:</span><span class="hspace">&nbsp;</span><span class="RktCmt">this</span><span class="hspace">&nbsp;</span><span class="RktCmt">is</span><span class="hspace">&nbsp;</span><span class="RktCmt">C</span><span class="hspace">&nbsp;</span><span class="RktCmt">code</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_create</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_surface_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*target</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a <span class="stt">cairo_surface_t</span> and +returns a pointer to a <span class="stt">cairo_t</span>. Let&rsquo;s start with a very simple C type +that matches up with this behavior: </div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span></p></div></div> + +<div class="SIntrapara">This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type.</div> + +<p>The following definition shows how to use this type to obtain a foreign +function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Then we can use <span class="RktWrap"><span class="RktSym">cairo-create</span></span> as an ordinary racket function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">ctx</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Interlude__more_type_safety)"></a>Interlude: more type safety</h1> + +<p>Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> types, it is easy +to accidentally misuse the function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">You may not want to actually run this</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a cairo_t is not a cairo_surface_t</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>To prevent such bad uses, it is good practice to use <span class="emph">tagged</span> pointer types +using <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>. Here are two example definitions that +correspond to the <span class="stt">cairo_t</span> and <span class="stt">cairo_surface_t</span> types from earlier:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">The leading underscores are mandatory</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>We can then redefine <span class="RktWrap"><span class="RktSym">cairo-create</span></span> with a better type, which will +prevent ill-typed calls:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">cairo_surface_t-&gt;C: argument is not non-null</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">`cairo_surface_t' pointer</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktErr">argument: #&lt;cpointer:cairo_t&gt;</span></p></td></tr></tbody></table></div> + +<p>Unfortunately our old definition of <span class="RktWrap"><span class="RktSym">ctx</span></span> doesn&rsquo;t have this tag:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#f</span></p></td></tr></tbody></table></div> + +<p>Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it&rsquo;s also possible to update +existing pointers with a tag like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-push-tag%21%29%29">cpointer-push-tag!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#t</span></p></td></tr></tbody></table></div> + +<p>Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).</p> + +<h1><a name="(part._.Macros_for_reducing_boilerplate)"></a>Macros for reducing boilerplate</h1> + +<p>Now let&rsquo;s start building the FFI bindings for the functions in the Cairo sample. +First, let&rsquo;s go ahead and look at all of the types for the sample functions +from the C API docs:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_cap_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">line_cap</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Starting with <span class="stt">cairo_move_to</span>, we can set up a definition like we did +with <span class="stt">cairo_create</span> before:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">"cairo_move_to"</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span> library that help reduce the +verbosity. Here&rsquo;s an alternative definition of <span class="RktWrap"><span class="RktSym">cairo-move-to</span></span> +using the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form from +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>As you can see, the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +<span class="RktWrap"><span class="RktSym">cairo_move_to</span></span>) you also don&rsquo;t need to supply the C name either.</p> + +<p>The definitions for <span class="stt">cairo_line_to</span>, <span class="stt">cairo_set_line_width</span>, and +<span class="stt">cairo_stroke</span> aren&rsquo;t very interesting, so I&rsquo;ll just include them +below without comment:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="stt">cairo_set_line_cap</span> case is more interesting because the type +<span class="stt">cairo_line_cap_t</span> is a C enumeration type. Racket&rsquo;s FFI comes with +convenience forms for defining enumeration types&#8212; + <wbr />though it&rsquo;s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here&rsquo;s a quote from the Barzilay and Orlovsky +paper on that:</p> + +<blockquote class="SubFlow"> + <p>Our design follows a simple principle: keep C-level +functionality to a minimum.</p></blockquote> + +<p>and specifically about enumerations:</p> + +<blockquote class="SubFlow"> + <p>For example, the C level part of our interface does not commit to a +specific implementation for enumerations &#8212; it simply exposes C integers.</p></blockquote> + +<p>To define an enumeration, we can use the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the <span class="stt">cairo_line_cap_t</span> type, it suffices to +just supply the cases as a list of symbols:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It&rsquo;s also possible to specify how the symbols map to integers more precisely +(see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> for those details).</p> + +<p>Given this type, we can specify the type for the line cap function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<h1><a name="(part._.Putting_it_all_together)"></a>Putting it all together</h1> + +<p>Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">30.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-cap</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">round</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object <span class="RktWrap"><span class="RktSym">bt</span></span>. To see the +results of what we drew, we can just evaluate <span class="RktWrap"><span class="RktSym">bt</span></span> at the REPL. But it&rsquo;s +a little nicer if we use the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> library to draw a frame around +it to distinguish it from the background:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>And we&rsquo;re done! Of course, there is a lot more to the FFI. For example, I haven&rsquo;t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I&rsquo;m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!</p> \ No newline at end of file diff --git a/blog/feeds/Gradual-Typing.atom.xml b/blog/feeds/Gradual-Typing.atom.xml new file mode 100644 index 00000000..14750814 --- /dev/null +++ b/blog/feeds/Gradual-Typing.atom.xml @@ -0,0 +1,32 @@ + + + PRL Blog: Posts tagged 'Gradual Typing' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Gradual-Typing-html + 2017-05-09T14:04:31Z + + No Good Answers, Gradually Typed Object-Oriented Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-05-09-no-good-answers-gradually-typed-object-oriented-languages + 2017-05-09T14:04:31Z + 2017-05-09T14:04:31Z + + Ben Chung + <!-- more--> + +<p>Untyped code remains a real problem in practice, as a result of reduced performance and hindered readability. One approach to solve this problem is gradual typing.</p> + +<p>Gradual typing puts the onus on the developer to add type annotations, statically checks whatever type annotations have been written, and dynamically ensures that untyped code does not violate those annotations. A number of approaches have been put forward to try to achieve these objectives while retaining efficiency, semantic meaning, and the ability to actually type untyped code.</p> + +<p>I discussed three systems, all of which achieve the objective of typing untyped code in different ways, and all of which have different tradeoffs.</p> + +<p>Unofficial Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md</a></li></ul> + +<p>Code Examples:</p> + +<ul> + <li><a href="https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL">https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Gradual-Typing.rss.xml b/blog/feeds/Gradual-Typing.rss.xml new file mode 100644 index 00000000..d2786220 --- /dev/null +++ b/blog/feeds/Gradual-Typing.rss.xml @@ -0,0 +1,32 @@ + + + + PRL Blog: Posts tagged 'Gradual Typing' + PRL Blog: Posts tagged 'Gradual Typing' + http://prl.ccs.neu.edu/blog/tags/Gradual-Typing.html + Tue, 09 May 2017 14:04:31 UT + Tue, 09 May 2017 14:04:31 UT + 1800 + + No Good Answers, Gradually Typed Object-Oriented Languages + http://prl.ccs.neu.edu/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/?utm_source=Gradual-Typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-09-no-good-answers-gradually-typed-object-oriented-languages + Tue, 09 May 2017 14:04:31 UT + Ben Chung + <!-- more--> + +<p>Untyped code remains a real problem in practice, as a result of reduced performance and hindered readability. One approach to solve this problem is gradual typing.</p> + +<p>Gradual typing puts the onus on the developer to add type annotations, statically checks whatever type annotations have been written, and dynamically ensures that untyped code does not violate those annotations. A number of approaches have been put forward to try to achieve these objectives while retaining efficiency, semantic meaning, and the ability to actually type untyped code.</p> + +<p>I discussed three systems, all of which achieve the objective of typing untyped code in different ways, and all of which have different tradeoffs.</p> + +<p>Unofficial Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md</a></li></ul> + +<p>Code Examples:</p> + +<ul> + <li><a href="https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL">https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/HOPL.atom.xml b/blog/feeds/HOPL.atom.xml new file mode 100644 index 00000000..5ef3bf13 --- /dev/null +++ b/blog/feeds/HOPL.atom.xml @@ -0,0 +1,502 @@ + + + PRL Blog: Posts tagged 'HOPL' + + + urn:http-prl-ccs-neu-edu:-blog-tags-HOPL-html + 2017-05-09T14:04:31Z + + No Good Answers, Gradually Typed Object-Oriented Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-05-09-no-good-answers-gradually-typed-object-oriented-languages + 2017-05-09T14:04:31Z + 2017-05-09T14:04:31Z + + Ben Chung + <!-- more--> + +<p>Untyped code remains a real problem in practice, as a result of reduced performance and hindered readability. One approach to solve this problem is gradual typing.</p> + +<p>Gradual typing puts the onus on the developer to add type annotations, statically checks whatever type annotations have been written, and dynamically ensures that untyped code does not violate those annotations. A number of approaches have been put forward to try to achieve these objectives while retaining efficiency, semantic meaning, and the ability to actually type untyped code.</p> + +<p>I discussed three systems, all of which achieve the objective of typing untyped code in different ways, and all of which have different tradeoffs.</p> + +<p>Unofficial Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md</a></li></ul> + +<p>Code Examples:</p> + +<ul> + <li><a href="https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL">https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL</a></li></ul> + + Categorical Semantics for Dynamically Typed Programming Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + 2017-05-01T12:25:17Z + 2017-05-01T12:25:17Z + + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> + + What is Soft Typing? + + urn:http-prl-ccs-neu-edu:-blog-2017-04-28-what-is-soft-typing + 2017-04-28T12:25:17Z + 2017-04-28T12:25:17Z + + Ben Greenman + <!-- more--> + +<p>A soft type system rewrites programs and meets a few <em>design criteria</em>.</p> + +<hr /> + +<h2 id="what-are-the-design-criteria">What are the Design Criteria?</h2> + +<p>According to Mike Fagan&rsquo;s 1991 <a href="https://github.com/nuprl/hopl-s2017/tree/master/soft-typing/papers">dissertation</a>, a soft type system must:</p> + +<ul> + <li>accept all <em>syntactically correct</em> programs as input;</li> + <li>produce equivalent, <em>memory-safe</em> programs as output; and</li> + <li>be <em>unobtrusive</em></li></ul> + +<h3 id="important-details">Important details:</h3> + +<ul> + <li>In this context, <em>memory safe</em> basically means &ldquo;no segfaults&rdquo;. Programs output by a soft type system should be as safe as statically-typed Java programs or dynamically-typed Python programs.</li> + <li>Fagan characterizes <em>unobtrusive</em> with two general principles:</li> + <li><em>minimal text principle</em> : the type checker should work without any programmer-supplied annotations</li> + <li><em>minimal failure principle</em> : the type checker should assign <em>useful</em> types to <em>idiomatic</em> programs (basically, don&rsquo;t just say that every expression has &ldquo;unknown&rdquo; or &ldquo;top&rdquo; type)</li></ul> + +<h2 id="why-would-i-want-to-use-a-soft-type-system">Why would I want to use a soft type system?</h2> + +<p>If you:</p> + +<ul> + <li>like dynamic typing</li> + <li>want some <em>benefits</em> of static typing</li> + <li>refuse to (or <em>cannot</em>!) change your code to satisfy a type checker</li></ul> + +<p>then Soft Typing is a perfect fit. You just need to find/build a soft type checker.</p> + +<h3 id="clarification">Clarification</h3> + +<p>The <em>benefits</em> of static typing that a soft type system can give are:</p> + +<ul> + <li>early detection of typos and simple logical errors</li> + <li>documentation, through (inferred) type signatures</li> + <li>speed, when the types can justify removing a runtime safety check</li></ul> + +<p>See Andrew Wright&rsquo;s 1994 <a href="https://github.com/nuprl/hopl-s2017/tree/master/soft-typing/papers">dissertation</a> for proof.</p> + +<h2 id="can-i-use-andrew-wrights-soft-type-system">Can I use Andrew Wright&rsquo;s soft type system?</h2> + +<p>Not sure, but you may download the code for it:</p> + +<ul> + <li><a href="https://github.com/nuprl/softscheme">https://github.com/nuprl/softscheme</a></li></ul> + +<h2 id="please-explain-fagans--wrights-soft-types">Please explain Fagan&rsquo;s / Wright&rsquo;s soft types</h2> + +<p>Types <code>t</code> are made of constructors <code>k</code>, flags <code>f</code>, and type variables <code>a</code>. The grammar for types is basically:</p> + +<pre><code> t ::= a | (k f t ...) U t + k ::= Int | Pair | -&gt; + f ::= ++ | -- | b + a ::= a0 | a1 | a2 | a3 | .... + b ::= b0 | b1 | b2 | b3 | ....</code></pre> + +<p>where:</p> + +<ul> + <li><code>U</code> is just a symbol, represents &ldquo;union&rdquo;</li> + <li><code>a</code> is a type variable; there are infinitely many type variables</li> + <li><code>b</code> is a flag variable; the set of flag variables is also infinte</li></ul> + +<p>There are also some rules for types to be well-formed.</p> + +<p>Here are two well-formed types:</p> + +<pre><code>(Int ++) U a0 + +(-&gt; ++ ((Int b0) U a1) + ((Int ++) U a2)) U a3</code></pre> + +<p>Here are two types that match the grammar, but are <strong>NOT</strong> well-formed:</p> + +<pre><code>(Int ++ a0) U a1 + +(-&gt; --) U a2</code></pre> + +<p>Finally, some intuition:</p> + +<ul> + <li>A constructor <code>k</code> is like a behavior,</li> + <li>a type <em>describes</em> the behaviors a value can have.</li> + <li>The description is like a bitvector of &ldquo;yes&rdquo;, &ldquo;no&rdquo;, or &ldquo;maybe&rdquo; for each possible behavior.</li> + <li>A flag variable is the way to say &ldquo;maybe&rdquo;.</li> + <li>Every type ends with a type variable because every typed expression might flow to a context that expects a more general type.</li></ul> + +<p>The type and flag variables let Fagan and Wright encode subtyping using polymorphism. It&rsquo;s a very cool idea, introduced in Didier Remy&rsquo;s <a href="https://github.com/nuprl/hopl-s2017/tree/master/soft-typing/papers">POPL 1989 paper</a>. But it adds a learning curve, and has some drawbacks for type inference.</p> + +<h2 id="stream-of-consciousness-notes-from-the-hopl-lecture">Stream-of-consciousness notes from the HOPL lecture</h2> + +<ul> + <li><a href="/blog/static/soft-typing.pdf">Local copy</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/soft-typing">Source of Truth</a></li></ul> + + Refinement Types + + urn:http-prl-ccs-neu-edu:-blog-2017-04-20-refinement-types + 2017-04-20T23:38:23Z + 2017-04-20T23:38:23Z + + Kevin Clancy + <!-- more--> + +<p>Roughly, a refinement type system is an extra layer of precision, enforced through subtyping, added onto an existing type system. A base type is decomposed into a set of <em>base refinements</em>, each of which denotes a subset of the values belonging to the base type. A subtyping relation respecting set inclusion can then be defined among the refinements of the base type. These subtyping relations can be lifted onto a subtyping relation for compound types using a standard arrow subtyping rule.</p> + +<p>Extra type-checking precision sounds great, but what in practical terms does this precision look like? Freeman and Pfenning&rsquo;s &rsquo;92 paper <em>Refinement Types for ML</em> proposes extending ML&rsquo;s type definition language with constructs for decomposing a discriminated union type into a lattice of subtypes. For example, it allows the decomposition of a list type into a lattice including base refinements for empty lists, non-empty lists, and singletons. Those with experience in functional programming will realize this alleviates the dreaded and inescapable “non-exhaustive pattern match” warning, which tends to crop up in situations where the programmer understands that an exhaustive pattern match is not necessary.</p> + +<p>In the late 90&rsquo;s Xi and Pfenning advanced the state of refinement types by introducing a dependent refinement type system, implemented as a tool called Dependent ML. Their approach identifies a base refinement using a tuple of terms drawn from some computationally tractable constraint language called an <em>index language</em>. A list datatype can then be refined with a term of the <em>linear integer arithmetic</em> index language, denoting the subset of all lists having a specific length. One list refinement is then considered a subtype of another when a constraint solver can prove their index terms equal. Vazou et. al.&rsquo;s recent project Liquid Haskell is another dependent refinement type system which decides subtyping among base types by invoking an SMT solver under a context-dependent set of constraints. It differs significantly from Dependent ML in that it refines base types with certain well-behaved program terms rather than indices.</p> + +<hr /> + +<p>Resources:</p> + +<ul> + <li><a href="/blog/static/refinement_types_lecture.pdf">Full Notes</a></li> + <li><a href="/blog/static/refinement_types_bib.pdf">Annotated Bibliography</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/refinement-types">GitHub</a></li></ul> + + Type-Directed Compilation, Parts I and II + + urn:http-prl-ccs-neu-edu:-blog-2017-04-17-type-directed-compilation-parts-i-and-ii + 2017-04-17T12:00:17Z + 2017-04-17T12:00:17Z + Leif Andersen + William J. Bowman + + Leif Andersen, William J. Bowman + <!-- more--> + +<h3 id="part-i-type-directed-compilation-by-leif-andersen">Part I: <em>Type-Directed Compilation</em>, by Leif Andersen.</h3> + +<p>In this talk we discuss the history of type directed compilation. We start with Xavier Leroy&rsquo;s seminal paper: <a href="http://gallium.inria.fr/~xleroy/publi/unboxed-polymorphism.pdf"><em>Unboxed Objects and Polymorphic Typing</em></a>, continue to <a href="https://www.cs.cmu.edu/~rwh/papers/til/pldi96.pdf">TIL</a> (Typed Intermediate Language), and finish up with <a href="https://dash.harvard.edu/handle/1/2797451">TAL</a> (Typed Assembly Language). We talk about what it means for a compiler to be typed preserving, and give examples of optimizations that are enabled by types.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-24.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;24.md</a></li></ul> + +<h3 id="part-ii-dependent-type-directed-compilation-by-william-j-bowman">Part II: <em>Dependent Type-Directed Compilation</em>, by William J. Bowman</h3> + +<p>A certifying compiler is not verified, but it produces a proof of correctness for each binary. This proof can be independently checked to show that the binary was compiled correctly, removing the compiler from the trusted code base. Certifying compilation has its roots in preserving type-preserving compilation, and in particular in preserving dependent types. We start the history of dependent-type-preserving compilation with a compiler from C to Assembly. We&rsquo;ll see a result showing that preserving dependent types isn&rsquo;t possible, and then we&rsquo;ll do it anyway.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-28.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;28.md</a></li></ul> + +<p>Notes (to appear here, eventually):</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation">https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation</a></li></ul> + + Top Five Results of the Past 50 Years of Programming Languages Research + + urn:http-prl-ccs-neu-edu:-blog-2017-04-04-top-five-results-of-the-past-50-years-of-programming-languages-research + 2017-04-04T10:21:36Z + 2017-04-04T10:21:36Z + + Ben Greenman + +<p>Over the past 50 years, which result from programming languages research has had the greatest impact on working programmers?</p> +<!-- more--> + +<p>The center of the universe for a working programmer is the language (or languages) they use. Fundamental results in programming languages (PL) research can re-shape this universe.</p> + +<p>In <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">HOPL</a> two weeks ago, Matthias claimed that <em>type soundness</em> is the most useful and influential result to flow from PL research to PL practice in the last 50 years.</p> + +<p>But 50 years is a long time, and there are many serious contenders for the title of <em>greatest PL result</em>. Here are my (alphabetized) picks for the top five:</p> + +<h3 id="abstraction">Abstraction</h3> + +<blockquote> + <p>My goal in library design is this; I want to have a precise, elegant, re-usable abstraction &mdash;Conal Eliott, <a href="https://www.youtube.com/watch?v=zzCrZEil9iI">BayHac 2014 (00:01:55)</a></p></blockquote> + +<p>By <em>abstraction</em>, I mean anything whose interface is not just &ldquo;read the implementation&rdquo;. Could be a tuple, module, object, structure, semaphore, macro, etc. Even the memory hierarchy pyramid in your operating systems textbook is an abstraction. They are everywhere, and they are what separates computer science (it&rsquo;s about <em>ideas</em>) from electrical engineering (it&rsquo;s about <em>transistors</em>). Thank you <a href="/img/l-plp-1965.pdf">Peter Landin</a> and <a href="/img/m-thesis-1969.pdf">J.H. Morris</a>.</p> + +<h3 id="generational-garbage-collection">Generational Garbage Collection</h3> + +<p>I don&rsquo;t know much about garbage collection. I do know that I want it, and I&rsquo;m pretty sure that I wouldn&rsquo;t have it (outside of research languages) without generational garbage collection. Thank you <a href="/img/u-sde-1984.pdf">David Ungar</a>.</p> + +<h3 id="generic-programming">Generic Programming</h3> + +<p><em>a.k.a. the mainstream interpretations of parametric polymorphism</em></p> + +<p>The thought of programming in Java 1.4 is terrifying. Thank you <a href="/img/g-thesis-1972.pdf">Jean-Yves Girard</a> and <a href="/img/r-cp-1974.pdf">John C. Reynolds</a> and <a href="http://homepages.inf.ed.ac.uk/wadler/gj/">Gilad Bracha and Martin Odersky and David Stoutamire and Philip Wadler</a>.</p> + +<h3 id="modularization">Modularization</h3> + +<p>How can humans understand large software systems? By organizing the systems into smaller components (modules, objects) with well-defined interfaces. It&rsquo;s hard to imagine, but once upon a time the question of <em>how</em> to divide a system into modules was a new research problem. Thank you <a href="/img/p-tr-1971.pdf">D.L. Parnas</a>.</p> + +<h3 id="type-soundness">Type Soundness</h3> + +<p>Let me make two modest claims:</p> + +<ul> + <li>Soundness (with respect to a dynamic semantics) is a desirable property for a static type system.</li> + <li>A large number (at least, thousands) of working programmers agree that programming in a language with a sound, static type system is &ldquo;a good thing&rdquo;.</li></ul> + +<p>Neither of these claims were true 50 years ago. They are definitely true today. And the slogan &ldquo;well typed programs do not go wrong (up to a well-defined set of runtime errors)&rdquo; has become the catchphrase of PL research. Thank you <a href="/img/m-jcss-1978.pdf">Robin Milner</a>.</p> + +<h2 id="honorable-mentions">Honorable Mentions</h2> + +<ul> + <li><a href="http://www.paulgraham.com/thist.html">lexical scope</a></li> + <li><a href="/img/d-thesis-1984.pdf">type inference</a></li> + <li><a href="https://www.quora.com/Why-does-Kent-Beck-refer-to-the-rediscovery-of-test-driven-development">test-driven development</a></li> + <li><a href="https://en.wikipedia.org/wiki/Simula">object-oriented programming</a></li> + <li><a href="/img/ss-tr-1975.pdf">continuation passing style</a></li> + <li><a href="/img/kffd-tr-1986.pdf">hygienic macros</a></li></ul> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/">Gradual Typing Across the Spectrum</a></li> + <li><a href="http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/">Getting Started in Programming Languages</a></li> + <li><a href="https://williamjbowman.com/blog/2017/03/24/what-even-is-compiler-correctness/">What even is compiler correctness?</a></li></ul> + + Tracing JITs for Dynamic Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-03-15-tracing-jits-for-dynamic-languages + 2017-03-15T10:54:39Z + 2017-03-15T10:54:39Z + + Ming-Ho Yee + <!-- more--> + +<p>Traditional JIT (just-in-time) compilers are method-based: they compile &ldquo;hot&rdquo; (i.e. frequently executed) methods to native code. An alternative is trace-based or tracing JITs, where the compilation unit is a (hot) sequence of instructions. Typically, such sequences of instructions correspond to loops, where programs spend most of their execution time.</p> + +<p>Where did the idea of tracing come from? What was appealing about it? How was tracing adapted for JITs and dynamic languages? What happened to Mozilla&rsquo;s TraceMonkey, which used to be part of Firefox? Do any JITs today use tracing?</p> + +<p>In this talk, I trace tracing JITs from their origins to some of their recent developments. I cover five papers: the original tracing paper, an implementation of a tracing JIT for Java, the TraceMonkey JIT for JavaScript, PyPy&rsquo;s &ldquo;meta-level&rdquo; tracing, and a specific class of optimizations for tracing JITs.</p> + +<p><em>(The idea of using the phrase &ldquo;trace tracing JITs&rdquo; is from Matthias Felleisen.)</em></p> + +<p>All materials can be found in the <a href="https://github.com/nuprl/hopl-s2017/tree/master/tracing-jit">course repository</a>:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/tracing-jit/notes.pdf">Full notes</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/tracing-jit/annotated.txt">Annotated bibliography</a></li></ul> + +<hr /> + +<p><em>If you liked this post, you may also be interested in <a href="http://prl.ccs.neu.edu/blog/2019/01/28/on-stack-replacement/">on-stack replacement</a>.</em></p> + + Type Inference in Stack-Based Programming Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-03-10-type-inference-in-stack-based-programming-languages + 2017-03-10T16:23:30Z + 2017-03-10T16:23:30Z + + Rob Kleffner + <!-- more--> + +<p>Stack-based languages occupy a niche in today&rsquo;s programming language environment. The predominant stack-based language in use by programmers is Forth, and is found mostly on embedded devices. These languages also find use as compile targets for more popular languages: the CIL and JVM are both stack-based. Less popular but highly interesting languages to mention include <a href="http://www.kevinalbrecht.com/code/joy-mirror/joy.html">Joy</a> and <a href="http://factorcode.org/">Factor</a>, known for their emphasis on higher-order stack-based programming.</p> + +<p>The majority of stack-based languages are not statically typed, and it would be a stretch to call Forth even dynamically typed. As such, developing large projects in Forth or Factor can require great discipline on the part of the programmer to avoid type errors.</p> + +<p>In this talk, I presented the development of type inference for stack-based languages as a linear sequence, divided into two overarching segments:</p> + +<ul> + <li>An algebraic system known as <em>stack effects</em></li> + <li>Systems that can be encoded as <em>nested pairs</em> in standard functional programming languages</li></ul> + +<p>The thread of research on stack effects began with Jaanus Pöial in the early 1990&rsquo;s, and is a formalization of a commenting style well-known in the Forth community. The nested tuple systems were first examined by Okasaki in 1993 in the context of Haskell, and were later applied to higher-order stack-based languages. At the end, I give some avenues for extending the research on these systems, and list some pitfalls to be avoided in further research.</p> + +<p>Full notes (as PDF documents) &mdash; see the <a href="https://github.com/nuprl/hopl-s2017/tree/master/type-inference-for-stack-languages">git repository</a> for more documents:</p> + +<ul> + <li><a href="/blog/static/stack-languages-talk-notes.pdf">Talk notes</a></li> + <li><a href="/blog/static/stack-languages-annotated-bib.pdf">Annotated bibliography</a></li></ul> + + Linear Types for Low-level Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-linear-types-for-low-level-languages + 2017-02-28T09:51:55Z + 2017-02-28T09:51:55Z + + Daniel Patterson + <!-- more--> + +<p>In this talk, we covered early papers (primarily, by Girard, Lafont, and Abramsky) on linear logic and its reflections into computation. The goal was to understand why linearity is often turned to as a principled way to control resource usage, as shows up in a language like Rust. From the very beginning, researchers realized the implications for &ldquo;low-level&rdquo; languages - that linear resources would eliminate the need for garbage collection, allow in-place mutation, and enable safe parallel computation. However, pure implementations of linearity incur lots of copying, doing away with any efficiency gained, and we covered a survey of papers that attempted to reconcile this contradiction by weakening linearity in controlled ways.</p> + +<p>Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-02-14.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;02&ndash;14.md</a></li></ul> + +<hr /> + +<p>Just after the talk, over lunch, we had a lab discussion about the phrase &ldquo;low level&rdquo;. Here are some thoughts:</p> + +<ul> + <li>the phrase is relative, both over time and depending on the programming task at hand</li> + <li>a &ldquo;low level&rdquo; task is &ldquo;one that you shouldn&rsquo;t need to worry about&rdquo; while solving your current task</li></ul> + +<p>And here are some example &ldquo;low-level&rdquo; tasks:</p> + +<ul> + <li>Time and space management is &ldquo;low level&rdquo; when designing a new algorithm (the first question is correctness)</li> + <li>Calling conventions and endian-ness (facets of the damn machine running the programs) are almost always low-level</li> + <li>Whether a given value is serializable is usually low-level</li> + <li>Possible side effects, thrown exceptions, and optional arguments can all be considered &ldquo;low level&rdquo; aspects of library functions. This is low-level in the sense that &ldquo;I&rsquo;d rather use a simpler type to think about this library&rdquo;</li> + <li>Managing type annotations is a low-level detail in ML programs</li></ul> + + Datalog for Static Analysis + + urn:http-prl-ccs-neu-edu:-blog-2017-02-21-datalog-for-static-analysis + 2017-02-21T12:58:27Z + 2017-02-21T12:58:27Z + + Ben Greenman + <!-- more--> + +<p>Datalog is an old DSL that frequently appears in work on static analysis. This edition of <a href="/blog/2017/02/15/introducing-hopl-2017/">HOPL 2017</a> explores the origins of Datalog in general, its early use in program analysis, and why Datalog remains a useful tool.</p> + +<p>Full notes:</p> + +<ul> + <li><a href="/blog/static/datalog-for-static-analysis.pdf">Local Copy</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/datalog-for-static-analysis">Source of Truth</a></li></ul> + +<hr /> + +<p>Datalog as a language was introduced by 1978 (its semantic foundations date back to 1976). It is <em>predicate logic</em> as a database query language. The traditional view of a Datalog program is a <em>time invariant</em> transformation over the <em>time varying</em> data stored in an external database.</p> + +<p>In the early 1990&rsquo;s, Uwe Aβmann designed a graph rewriting systems (EARS) that could:</p> + +<ol> + <li>Uniformly express various problems in static analysis</li> + <li>Systematically derive efficient solutions to such problems.</li></ol> + +<p>(Prior work had derived the same solutions with ad-hoc methods.) Aβmann&rsquo;s system is equivalent to Datalog.</p> + +<p>In 1993, Reps used the + <tt>CORAL</tt> deductive database (an implementation of Datalog) to derive an on-demand (read: lazy) implementation of program slicing from a <em>specification</em> of the slicing problem.</p> + +<p>Both Aβmann&rsquo;s and Reps work appeared in 1994. This was the first time Datalog had been used to implement a static analysis.</p> + +<p>Researchers continue to use Datalog because:</p> + +<ul> + <li>predicate logic (specifically: Horn clauses without function symbols or negation) is useful for expressing recursive relations &hellip; and static analyses are all about recursive relations</li> + <li>the language separates <em>specifications</em> from their <em>implementation</em></li> + <li>there are many techniques for efficiently serving a Datalog query</li> + <li>these techniques have been implemented in <a href="https://developer.logicblox.com/wp-content/uploads/2016/01/logicblox-sigmod15.pdf">at least one</a> commercial Datalog engine</li></ul> + +<p>For an excellent description of how Datalog can benefit static analysis, see the introduction to <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.648.1834&amp;rep=rep1&amp;type=pdf">Rep&rsquo;s paper</a>.</p> + + Conversational Context and Concurrency + + urn:http-prl-ccs-neu-edu:-blog-2017-02-15-conversational-context-and-concurrency + 2017-02-15T01:21:55Z + 2017-02-15T01:21:55Z + + Tony Garnock-Jones + <!-- more--> + +<p>When programs are written with concurrency in mind, the programmer reasons about the interactions between concurrent components or agents in the program. This includes exchange of information, as well as management of resources, handling of partial failure, collective decision-making and so on.</p> + +<p>These components might be objects, or threads, or processes, or actors, or some more nebulous and loosely-defined concept; a group of callbacks, perhaps. The programmer has the notion of an agent in their mind, which translates into some representation of that agent in the program.</p> + +<p>We think about the contexts (because there can be more than one) in which agents exist in two different ways. From each agent&rsquo;s perspective, the important thing to think about is the boundary between the agent and everything else in the system. But from the system perspective, we often think about <em>conversations</em> between agents, whether it&rsquo;s just two having an exchange, or a whole group collaborating on some task. Agents in a conversation play different roles, join and leave the group, and build shared conversational state.</p> + +<p>In this talk, I used the idea of these <em>conversational contexts</em> as a lens through which to view the development of various metaphors and mechanisms of communication and coordination. I presented four <em>computational models</em> for concurrent interaction:</p> + +<ul> + <li>monitors, and shared memory concurrency generally</li> + <li>the actor model</li> + <li>channel-based communication</li> + <li>tuplespaces</li></ul> + +<p>These aren&rsquo;t full programming languages, but there are many <em>programming models</em> that build upon them. In some cases, development of these ideas has progressed all the way up to <em>system models</em> including user interaction and so forth.</p> + +<p>The linked lecture notes include informal sketches of reduction semantics for each of the four models, plus a handful of small examples to give a feel for them.</p> + +<p>Lecture Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/conversational-context-and-concurrency/index.md">https://github.com/nuprl/hopl-s2017/tree/master/conversational-context-and-concurrency/index.md</a></li></ul> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-01-31.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;01&ndash;31.md</a></li></ul> + + Introducing HOPL 2017 + + urn:http-prl-ccs-neu-edu:-blog-2017-02-15-introducing-hopl-2017 + 2017-02-15T01:21:37Z + 2017-02-15T01:21:37Z + + Ben Greenman + +<p>This semester at Northeastern, Matthias Felleisen is organizing the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">History of Programming Languages</a> seminar. Look for posts tagged <code>HOPL</code> for updates from the lectures.</p> +<!-- more--> + +<p>Once every 6 to 8 years (i.e., once every batch of Ph.D. students?), <a href="http://www.ccs.neu.edu/home/matthias">Matthias Felleisen</a> teaches History of Programming Languages. Nominally, the course is a seminar. But unlike a typical seminar course, weekly topics are not the technical details from a handful of papers. Rather:</p> + +<blockquote> + <p>The primary goal is to understand (some of) the discipline as it exists today and how some of its major themes evolved.</p></blockquote> + +<blockquote> + <p>The secondary goal is to develop basic skills for understanding and describing research themes. Every student will learn to study a theme via a series of papers, prepare an annotated bibliography, and present the key steps in the evolution of the theme.</p></blockquote> + +<p><strong>Themes</strong> is the operative word. To set the tone, this semester started with &ldquo;themes that NUPRL faculty members have developed over the many decades of their careers.&rdquo;</p> + +<ul> + <li>Matthias, <em>Full Abstraction: From PCF to SPCF</em></li> + <li>Jan Vitek, <em>From Encapsulation to Ownership</em></li> + <li>Will Clinger, <em>Garbage Collection vs. Manual Allocation</em></li> + <li>Olin Shivers, <em>Higher-order Flow Analysis</em></li> + <li>Amal Ahmed, <em>Logical Relations: Stepping Beyond Toy Languages</em></li> + <li>Matthias, <em>Programming Languages and Calculi</em></li> + <li>Jan-Willem van de Meent, <em>Rescoring Strategies for Probabilistic Programs</em></li> + <li>(upcoming) Mitch Wand, <em>Analysis-Based Program Transformation</em></li> + <li>(upcoming) Frank Tip, <em>Refactoring</em></li></ul> + +<p>At this point in the course, we are just starting with the student presentations. As these presentations happen, we plan to push updates to this blog. All presentation materials are in the course repository:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017">https://github.com/nuprl/hopl-s2017</a></li></ul> + +<p>Speakers&rsquo; notes and annotated bibliographies are in top-level folders in the repo. Discussion summaries and &ldquo;unofficial&rdquo; notes are in the top-level <a href="https://github.com/nuprl/hopl-s2017/tree/master/lecture_notes"><code>lecture_notes/</code></a> folder.</p> + +<p>The list of upcoming presentations is online (along with <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/Summary___Materials.html">the papers</a> each presentation is based on):</p> + +<ul> + <li><a href="http://www.ccs.neu.edu/home/matthias/7480-s17/lectures.html">http://www.ccs.neu.edu/home/matthias/7480-s17/lectures.html</a></li></ul> + +<p>Blogs posts for each talk should appear 2 weeks after the talk happens.</p> + +<hr /> + +<p>Links to past editions of HOPL:</p> + +<ul> + <li><a href="http://www.ccs.neu.edu/home/matthias/369-s10/index.html">Spring 2010</a></li> + <li><a href="http://www.ccs.neu.edu/home/matthias/369-s04/index.html">Spring 2004</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/HOPL.rss.xml b/blog/feeds/HOPL.rss.xml new file mode 100644 index 00000000..21fa41f7 --- /dev/null +++ b/blog/feeds/HOPL.rss.xml @@ -0,0 +1,478 @@ + + + + PRL Blog: Posts tagged 'HOPL' + PRL Blog: Posts tagged 'HOPL' + http://prl.ccs.neu.edu/blog/tags/HOPL.html + Tue, 09 May 2017 14:04:31 UT + Tue, 09 May 2017 14:04:31 UT + 1800 + + No Good Answers, Gradually Typed Object-Oriented Languages + http://prl.ccs.neu.edu/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-09-no-good-answers-gradually-typed-object-oriented-languages + Tue, 09 May 2017 14:04:31 UT + Ben Chung + <!-- more--> + +<p>Untyped code remains a real problem in practice, as a result of reduced performance and hindered readability. One approach to solve this problem is gradual typing.</p> + +<p>Gradual typing puts the onus on the developer to add type annotations, statically checks whatever type annotations have been written, and dynamically ensures that untyped code does not violate those annotations. A number of approaches have been put forward to try to achieve these objectives while retaining efficiency, semantic meaning, and the ability to actually type untyped code.</p> + +<p>I discussed three systems, all of which achieve the objective of typing untyped code in different ways, and all of which have different tradeoffs.</p> + +<p>Unofficial Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-18.md</a></li></ul> + +<p>Code Examples:</p> + +<ul> + <li><a href="https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL">https://github.com/BenChung/GradualComparison/tree/master/examples/HOPL</a></li></ul> + + Categorical Semantics for Dynamically Typed Programming Languages + http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + Mon, 01 May 2017 12:25:17 UT + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> + + What is Soft Typing? + http://prl.ccs.neu.edu/blog/2017/04/28/what-is-soft-typing/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-04-28-what-is-soft-typing + Fri, 28 Apr 2017 12:25:17 UT + Ben Greenman + <!-- more--> + +<p>A soft type system rewrites programs and meets a few <em>design criteria</em>.</p> + +<hr /> + +<h2 id="what-are-the-design-criteria">What are the Design Criteria?</h2> + +<p>According to Mike Fagan&rsquo;s 1991 <a href="https://github.com/nuprl/hopl-s2017/tree/master/soft-typing/papers">dissertation</a>, a soft type system must:</p> + +<ul> + <li>accept all <em>syntactically correct</em> programs as input;</li> + <li>produce equivalent, <em>memory-safe</em> programs as output; and</li> + <li>be <em>unobtrusive</em></li></ul> + +<h3 id="important-details">Important details:</h3> + +<ul> + <li>In this context, <em>memory safe</em> basically means &ldquo;no segfaults&rdquo;. Programs output by a soft type system should be as safe as statically-typed Java programs or dynamically-typed Python programs.</li> + <li>Fagan characterizes <em>unobtrusive</em> with two general principles:</li> + <li><em>minimal text principle</em> : the type checker should work without any programmer-supplied annotations</li> + <li><em>minimal failure principle</em> : the type checker should assign <em>useful</em> types to <em>idiomatic</em> programs (basically, don&rsquo;t just say that every expression has &ldquo;unknown&rdquo; or &ldquo;top&rdquo; type)</li></ul> + +<h2 id="why-would-i-want-to-use-a-soft-type-system">Why would I want to use a soft type system?</h2> + +<p>If you:</p> + +<ul> + <li>like dynamic typing</li> + <li>want some <em>benefits</em> of static typing</li> + <li>refuse to (or <em>cannot</em>!) change your code to satisfy a type checker</li></ul> + +<p>then Soft Typing is a perfect fit. You just need to find/build a soft type checker.</p> + +<h3 id="clarification">Clarification</h3> + +<p>The <em>benefits</em> of static typing that a soft type system can give are:</p> + +<ul> + <li>early detection of typos and simple logical errors</li> + <li>documentation, through (inferred) type signatures</li> + <li>speed, when the types can justify removing a runtime safety check</li></ul> + +<p>See Andrew Wright&rsquo;s 1994 <a href="https://github.com/nuprl/hopl-s2017/tree/master/soft-typing/papers">dissertation</a> for proof.</p> + +<h2 id="can-i-use-andrew-wrights-soft-type-system">Can I use Andrew Wright&rsquo;s soft type system?</h2> + +<p>Not sure, but you may download the code for it:</p> + +<ul> + <li><a href="https://github.com/nuprl/softscheme">https://github.com/nuprl/softscheme</a></li></ul> + +<h2 id="please-explain-fagans--wrights-soft-types">Please explain Fagan&rsquo;s / Wright&rsquo;s soft types</h2> + +<p>Types <code>t</code> are made of constructors <code>k</code>, flags <code>f</code>, and type variables <code>a</code>. The grammar for types is basically:</p> + +<pre><code> t ::= a | (k f t ...) U t + k ::= Int | Pair | -&gt; + f ::= ++ | -- | b + a ::= a0 | a1 | a2 | a3 | .... + b ::= b0 | b1 | b2 | b3 | ....</code></pre> + +<p>where:</p> + +<ul> + <li><code>U</code> is just a symbol, represents &ldquo;union&rdquo;</li> + <li><code>a</code> is a type variable; there are infinitely many type variables</li> + <li><code>b</code> is a flag variable; the set of flag variables is also infinte</li></ul> + +<p>There are also some rules for types to be well-formed.</p> + +<p>Here are two well-formed types:</p> + +<pre><code>(Int ++) U a0 + +(-&gt; ++ ((Int b0) U a1) + ((Int ++) U a2)) U a3</code></pre> + +<p>Here are two types that match the grammar, but are <strong>NOT</strong> well-formed:</p> + +<pre><code>(Int ++ a0) U a1 + +(-&gt; --) U a2</code></pre> + +<p>Finally, some intuition:</p> + +<ul> + <li>A constructor <code>k</code> is like a behavior,</li> + <li>a type <em>describes</em> the behaviors a value can have.</li> + <li>The description is like a bitvector of &ldquo;yes&rdquo;, &ldquo;no&rdquo;, or &ldquo;maybe&rdquo; for each possible behavior.</li> + <li>A flag variable is the way to say &ldquo;maybe&rdquo;.</li> + <li>Every type ends with a type variable because every typed expression might flow to a context that expects a more general type.</li></ul> + +<p>The type and flag variables let Fagan and Wright encode subtyping using polymorphism. It&rsquo;s a very cool idea, introduced in Didier Remy&rsquo;s <a href="https://github.com/nuprl/hopl-s2017/tree/master/soft-typing/papers">POPL 1989 paper</a>. But it adds a learning curve, and has some drawbacks for type inference.</p> + +<h2 id="stream-of-consciousness-notes-from-the-hopl-lecture">Stream-of-consciousness notes from the HOPL lecture</h2> + +<ul> + <li><a href="/blog/static/soft-typing.pdf">Local copy</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/soft-typing">Source of Truth</a></li></ul> + + Refinement Types + http://prl.ccs.neu.edu/blog/2017/04/20/refinement-types/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-04-20-refinement-types + Thu, 20 Apr 2017 23:38:23 UT + Kevin Clancy + <!-- more--> + +<p>Roughly, a refinement type system is an extra layer of precision, enforced through subtyping, added onto an existing type system. A base type is decomposed into a set of <em>base refinements</em>, each of which denotes a subset of the values belonging to the base type. A subtyping relation respecting set inclusion can then be defined among the refinements of the base type. These subtyping relations can be lifted onto a subtyping relation for compound types using a standard arrow subtyping rule.</p> + +<p>Extra type-checking precision sounds great, but what in practical terms does this precision look like? Freeman and Pfenning&rsquo;s &rsquo;92 paper <em>Refinement Types for ML</em> proposes extending ML&rsquo;s type definition language with constructs for decomposing a discriminated union type into a lattice of subtypes. For example, it allows the decomposition of a list type into a lattice including base refinements for empty lists, non-empty lists, and singletons. Those with experience in functional programming will realize this alleviates the dreaded and inescapable “non-exhaustive pattern match” warning, which tends to crop up in situations where the programmer understands that an exhaustive pattern match is not necessary.</p> + +<p>In the late 90&rsquo;s Xi and Pfenning advanced the state of refinement types by introducing a dependent refinement type system, implemented as a tool called Dependent ML. Their approach identifies a base refinement using a tuple of terms drawn from some computationally tractable constraint language called an <em>index language</em>. A list datatype can then be refined with a term of the <em>linear integer arithmetic</em> index language, denoting the subset of all lists having a specific length. One list refinement is then considered a subtype of another when a constraint solver can prove their index terms equal. Vazou et. al.&rsquo;s recent project Liquid Haskell is another dependent refinement type system which decides subtyping among base types by invoking an SMT solver under a context-dependent set of constraints. It differs significantly from Dependent ML in that it refines base types with certain well-behaved program terms rather than indices.</p> + +<hr /> + +<p>Resources:</p> + +<ul> + <li><a href="/blog/static/refinement_types_lecture.pdf">Full Notes</a></li> + <li><a href="/blog/static/refinement_types_bib.pdf">Annotated Bibliography</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/refinement-types">GitHub</a></li></ul> + + Type-Directed Compilation, Parts I and II + http://prl.ccs.neu.edu/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-04-17-type-directed-compilation-parts-i-and-ii + Mon, 17 Apr 2017 12:00:17 UT + Leif Andersen, William J. Bowman + <!-- more--> + +<h3 id="part-i-type-directed-compilation-by-leif-andersen">Part I: <em>Type-Directed Compilation</em>, by Leif Andersen.</h3> + +<p>In this talk we discuss the history of type directed compilation. We start with Xavier Leroy&rsquo;s seminal paper: <a href="http://gallium.inria.fr/~xleroy/publi/unboxed-polymorphism.pdf"><em>Unboxed Objects and Polymorphic Typing</em></a>, continue to <a href="https://www.cs.cmu.edu/~rwh/papers/til/pldi96.pdf">TIL</a> (Typed Intermediate Language), and finish up with <a href="https://dash.harvard.edu/handle/1/2797451">TAL</a> (Typed Assembly Language). We talk about what it means for a compiler to be typed preserving, and give examples of optimizations that are enabled by types.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-24.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;24.md</a></li></ul> + +<h3 id="part-ii-dependent-type-directed-compilation-by-william-j-bowman">Part II: <em>Dependent Type-Directed Compilation</em>, by William J. Bowman</h3> + +<p>A certifying compiler is not verified, but it produces a proof of correctness for each binary. This proof can be independently checked to show that the binary was compiled correctly, removing the compiler from the trusted code base. Certifying compilation has its roots in preserving type-preserving compilation, and in particular in preserving dependent types. We start the history of dependent-type-preserving compilation with a compiler from C to Assembly. We&rsquo;ll see a result showing that preserving dependent types isn&rsquo;t possible, and then we&rsquo;ll do it anyway.</p> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-03-28.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;03&ndash;28.md</a></li></ul> + +<p>Notes (to appear here, eventually):</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation">https://github.com/nuprl/hopl-s2017/blob/master/dependent-type-preserving-compilation</a></li></ul> + + Top Five Results of the Past 50 Years of Programming Languages Research + http://prl.ccs.neu.edu/blog/2017/04/04/top-five-results-of-the-past-50-years-of-programming-languages-research/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-04-04-top-five-results-of-the-past-50-years-of-programming-languages-research + Tue, 04 Apr 2017 10:21:36 UT + Ben Greenman + +<p>Over the past 50 years, which result from programming languages research has had the greatest impact on working programmers?</p> +<!-- more--> + +<p>The center of the universe for a working programmer is the language (or languages) they use. Fundamental results in programming languages (PL) research can re-shape this universe.</p> + +<p>In <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">HOPL</a> two weeks ago, Matthias claimed that <em>type soundness</em> is the most useful and influential result to flow from PL research to PL practice in the last 50 years.</p> + +<p>But 50 years is a long time, and there are many serious contenders for the title of <em>greatest PL result</em>. Here are my (alphabetized) picks for the top five:</p> + +<h3 id="abstraction">Abstraction</h3> + +<blockquote> + <p>My goal in library design is this; I want to have a precise, elegant, re-usable abstraction &mdash;Conal Eliott, <a href="https://www.youtube.com/watch?v=zzCrZEil9iI">BayHac 2014 (00:01:55)</a></p></blockquote> + +<p>By <em>abstraction</em>, I mean anything whose interface is not just &ldquo;read the implementation&rdquo;. Could be a tuple, module, object, structure, semaphore, macro, etc. Even the memory hierarchy pyramid in your operating systems textbook is an abstraction. They are everywhere, and they are what separates computer science (it&rsquo;s about <em>ideas</em>) from electrical engineering (it&rsquo;s about <em>transistors</em>). Thank you <a href="/img/l-plp-1965.pdf">Peter Landin</a> and <a href="/img/m-thesis-1969.pdf">J.H. Morris</a>.</p> + +<h3 id="generational-garbage-collection">Generational Garbage Collection</h3> + +<p>I don&rsquo;t know much about garbage collection. I do know that I want it, and I&rsquo;m pretty sure that I wouldn&rsquo;t have it (outside of research languages) without generational garbage collection. Thank you <a href="/img/u-sde-1984.pdf">David Ungar</a>.</p> + +<h3 id="generic-programming">Generic Programming</h3> + +<p><em>a.k.a. the mainstream interpretations of parametric polymorphism</em></p> + +<p>The thought of programming in Java 1.4 is terrifying. Thank you <a href="/img/g-thesis-1972.pdf">Jean-Yves Girard</a> and <a href="/img/r-cp-1974.pdf">John C. Reynolds</a> and <a href="http://homepages.inf.ed.ac.uk/wadler/gj/">Gilad Bracha and Martin Odersky and David Stoutamire and Philip Wadler</a>.</p> + +<h3 id="modularization">Modularization</h3> + +<p>How can humans understand large software systems? By organizing the systems into smaller components (modules, objects) with well-defined interfaces. It&rsquo;s hard to imagine, but once upon a time the question of <em>how</em> to divide a system into modules was a new research problem. Thank you <a href="/img/p-tr-1971.pdf">D.L. Parnas</a>.</p> + +<h3 id="type-soundness">Type Soundness</h3> + +<p>Let me make two modest claims:</p> + +<ul> + <li>Soundness (with respect to a dynamic semantics) is a desirable property for a static type system.</li> + <li>A large number (at least, thousands) of working programmers agree that programming in a language with a sound, static type system is &ldquo;a good thing&rdquo;.</li></ul> + +<p>Neither of these claims were true 50 years ago. They are definitely true today. And the slogan &ldquo;well typed programs do not go wrong (up to a well-defined set of runtime errors)&rdquo; has become the catchphrase of PL research. Thank you <a href="/img/m-jcss-1978.pdf">Robin Milner</a>.</p> + +<h2 id="honorable-mentions">Honorable Mentions</h2> + +<ul> + <li><a href="http://www.paulgraham.com/thist.html">lexical scope</a></li> + <li><a href="/img/d-thesis-1984.pdf">type inference</a></li> + <li><a href="https://www.quora.com/Why-does-Kent-Beck-refer-to-the-rediscovery-of-test-driven-development">test-driven development</a></li> + <li><a href="https://en.wikipedia.org/wiki/Simula">object-oriented programming</a></li> + <li><a href="/img/ss-tr-1975.pdf">continuation passing style</a></li> + <li><a href="/img/kffd-tr-1986.pdf">hygienic macros</a></li></ul> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/">Gradual Typing Across the Spectrum</a></li> + <li><a href="http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/">Getting Started in Programming Languages</a></li> + <li><a href="https://williamjbowman.com/blog/2017/03/24/what-even-is-compiler-correctness/">What even is compiler correctness?</a></li></ul> + + Tracing JITs for Dynamic Languages + http://prl.ccs.neu.edu/blog/2017/03/15/tracing-jits-for-dynamic-languages/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-03-15-tracing-jits-for-dynamic-languages + Wed, 15 Mar 2017 10:54:39 UT + Ming-Ho Yee + <!-- more--> + +<p>Traditional JIT (just-in-time) compilers are method-based: they compile &ldquo;hot&rdquo; (i.e. frequently executed) methods to native code. An alternative is trace-based or tracing JITs, where the compilation unit is a (hot) sequence of instructions. Typically, such sequences of instructions correspond to loops, where programs spend most of their execution time.</p> + +<p>Where did the idea of tracing come from? What was appealing about it? How was tracing adapted for JITs and dynamic languages? What happened to Mozilla&rsquo;s TraceMonkey, which used to be part of Firefox? Do any JITs today use tracing?</p> + +<p>In this talk, I trace tracing JITs from their origins to some of their recent developments. I cover five papers: the original tracing paper, an implementation of a tracing JIT for Java, the TraceMonkey JIT for JavaScript, PyPy&rsquo;s &ldquo;meta-level&rdquo; tracing, and a specific class of optimizations for tracing JITs.</p> + +<p><em>(The idea of using the phrase &ldquo;trace tracing JITs&rdquo; is from Matthias Felleisen.)</em></p> + +<p>All materials can be found in the <a href="https://github.com/nuprl/hopl-s2017/tree/master/tracing-jit">course repository</a>:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/tracing-jit/notes.pdf">Full notes</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/tracing-jit/annotated.txt">Annotated bibliography</a></li></ul> + +<hr /> + +<p><em>If you liked this post, you may also be interested in <a href="http://prl.ccs.neu.edu/blog/2019/01/28/on-stack-replacement/">on-stack replacement</a>.</em></p> + + Type Inference in Stack-Based Programming Languages + http://prl.ccs.neu.edu/blog/2017/03/10/type-inference-in-stack-based-programming-languages/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-03-10-type-inference-in-stack-based-programming-languages + Fri, 10 Mar 2017 16:23:30 UT + Rob Kleffner + <!-- more--> + +<p>Stack-based languages occupy a niche in today&rsquo;s programming language environment. The predominant stack-based language in use by programmers is Forth, and is found mostly on embedded devices. These languages also find use as compile targets for more popular languages: the CIL and JVM are both stack-based. Less popular but highly interesting languages to mention include <a href="http://www.kevinalbrecht.com/code/joy-mirror/joy.html">Joy</a> and <a href="http://factorcode.org/">Factor</a>, known for their emphasis on higher-order stack-based programming.</p> + +<p>The majority of stack-based languages are not statically typed, and it would be a stretch to call Forth even dynamically typed. As such, developing large projects in Forth or Factor can require great discipline on the part of the programmer to avoid type errors.</p> + +<p>In this talk, I presented the development of type inference for stack-based languages as a linear sequence, divided into two overarching segments:</p> + +<ul> + <li>An algebraic system known as <em>stack effects</em></li> + <li>Systems that can be encoded as <em>nested pairs</em> in standard functional programming languages</li></ul> + +<p>The thread of research on stack effects began with Jaanus Pöial in the early 1990&rsquo;s, and is a formalization of a commenting style well-known in the Forth community. The nested tuple systems were first examined by Okasaki in 1993 in the context of Haskell, and were later applied to higher-order stack-based languages. At the end, I give some avenues for extending the research on these systems, and list some pitfalls to be avoided in further research.</p> + +<p>Full notes (as PDF documents) &mdash; see the <a href="https://github.com/nuprl/hopl-s2017/tree/master/type-inference-for-stack-languages">git repository</a> for more documents:</p> + +<ul> + <li><a href="/blog/static/stack-languages-talk-notes.pdf">Talk notes</a></li> + <li><a href="/blog/static/stack-languages-annotated-bib.pdf">Annotated bibliography</a></li></ul> + + Linear Types for Low-level Languages + http://prl.ccs.neu.edu/blog/2017/02/28/linear-types-for-low-level-languages/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-linear-types-for-low-level-languages + Tue, 28 Feb 2017 09:51:55 UT + Daniel Patterson + <!-- more--> + +<p>In this talk, we covered early papers (primarily, by Girard, Lafont, and Abramsky) on linear logic and its reflections into computation. The goal was to understand why linearity is often turned to as a principled way to control resource usage, as shows up in a language like Rust. From the very beginning, researchers realized the implications for &ldquo;low-level&rdquo; languages - that linear resources would eliminate the need for garbage collection, allow in-place mutation, and enable safe parallel computation. However, pure implementations of linearity incur lots of copying, doing away with any efficiency gained, and we covered a survey of papers that attempted to reconcile this contradiction by weakening linearity in controlled ways.</p> + +<p>Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-02-14.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;02&ndash;14.md</a></li></ul> + +<hr /> + +<p>Just after the talk, over lunch, we had a lab discussion about the phrase &ldquo;low level&rdquo;. Here are some thoughts:</p> + +<ul> + <li>the phrase is relative, both over time and depending on the programming task at hand</li> + <li>a &ldquo;low level&rdquo; task is &ldquo;one that you shouldn&rsquo;t need to worry about&rdquo; while solving your current task</li></ul> + +<p>And here are some example &ldquo;low-level&rdquo; tasks:</p> + +<ul> + <li>Time and space management is &ldquo;low level&rdquo; when designing a new algorithm (the first question is correctness)</li> + <li>Calling conventions and endian-ness (facets of the damn machine running the programs) are almost always low-level</li> + <li>Whether a given value is serializable is usually low-level</li> + <li>Possible side effects, thrown exceptions, and optional arguments can all be considered &ldquo;low level&rdquo; aspects of library functions. This is low-level in the sense that &ldquo;I&rsquo;d rather use a simpler type to think about this library&rdquo;</li> + <li>Managing type annotations is a low-level detail in ML programs</li></ul> + + Datalog for Static Analysis + http://prl.ccs.neu.edu/blog/2017/02/21/datalog-for-static-analysis/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-21-datalog-for-static-analysis + Tue, 21 Feb 2017 12:58:27 UT + Ben Greenman + <!-- more--> + +<p>Datalog is an old DSL that frequently appears in work on static analysis. This edition of <a href="/blog/2017/02/15/introducing-hopl-2017/">HOPL 2017</a> explores the origins of Datalog in general, its early use in program analysis, and why Datalog remains a useful tool.</p> + +<p>Full notes:</p> + +<ul> + <li><a href="/blog/static/datalog-for-static-analysis.pdf">Local Copy</a></li> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/datalog-for-static-analysis">Source of Truth</a></li></ul> + +<hr /> + +<p>Datalog as a language was introduced by 1978 (its semantic foundations date back to 1976). It is <em>predicate logic</em> as a database query language. The traditional view of a Datalog program is a <em>time invariant</em> transformation over the <em>time varying</em> data stored in an external database.</p> + +<p>In the early 1990&rsquo;s, Uwe Aβmann designed a graph rewriting systems (EARS) that could:</p> + +<ol> + <li>Uniformly express various problems in static analysis</li> + <li>Systematically derive efficient solutions to such problems.</li></ol> + +<p>(Prior work had derived the same solutions with ad-hoc methods.) Aβmann&rsquo;s system is equivalent to Datalog.</p> + +<p>In 1993, Reps used the + <tt>CORAL</tt> deductive database (an implementation of Datalog) to derive an on-demand (read: lazy) implementation of program slicing from a <em>specification</em> of the slicing problem.</p> + +<p>Both Aβmann&rsquo;s and Reps work appeared in 1994. This was the first time Datalog had been used to implement a static analysis.</p> + +<p>Researchers continue to use Datalog because:</p> + +<ul> + <li>predicate logic (specifically: Horn clauses without function symbols or negation) is useful for expressing recursive relations &hellip; and static analyses are all about recursive relations</li> + <li>the language separates <em>specifications</em> from their <em>implementation</em></li> + <li>there are many techniques for efficiently serving a Datalog query</li> + <li>these techniques have been implemented in <a href="https://developer.logicblox.com/wp-content/uploads/2016/01/logicblox-sigmod15.pdf">at least one</a> commercial Datalog engine</li></ul> + +<p>For an excellent description of how Datalog can benefit static analysis, see the introduction to <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.648.1834&amp;rep=rep1&amp;type=pdf">Rep&rsquo;s paper</a>.</p> + + Conversational Context and Concurrency + http://prl.ccs.neu.edu/blog/2017/02/15/conversational-context-and-concurrency/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-15-conversational-context-and-concurrency + Wed, 15 Feb 2017 01:21:55 UT + Tony Garnock-Jones + <!-- more--> + +<p>When programs are written with concurrency in mind, the programmer reasons about the interactions between concurrent components or agents in the program. This includes exchange of information, as well as management of resources, handling of partial failure, collective decision-making and so on.</p> + +<p>These components might be objects, or threads, or processes, or actors, or some more nebulous and loosely-defined concept; a group of callbacks, perhaps. The programmer has the notion of an agent in their mind, which translates into some representation of that agent in the program.</p> + +<p>We think about the contexts (because there can be more than one) in which agents exist in two different ways. From each agent&rsquo;s perspective, the important thing to think about is the boundary between the agent and everything else in the system. But from the system perspective, we often think about <em>conversations</em> between agents, whether it&rsquo;s just two having an exchange, or a whole group collaborating on some task. Agents in a conversation play different roles, join and leave the group, and build shared conversational state.</p> + +<p>In this talk, I used the idea of these <em>conversational contexts</em> as a lens through which to view the development of various metaphors and mechanisms of communication and coordination. I presented four <em>computational models</em> for concurrent interaction:</p> + +<ul> + <li>monitors, and shared memory concurrency generally</li> + <li>the actor model</li> + <li>channel-based communication</li> + <li>tuplespaces</li></ul> + +<p>These aren&rsquo;t full programming languages, but there are many <em>programming models</em> that build upon them. In some cases, development of these ideas has progressed all the way up to <em>system models</em> including user interaction and so forth.</p> + +<p>The linked lecture notes include informal sketches of reduction semantics for each of the four models, plus a handful of small examples to give a feel for them.</p> + +<p>Lecture Notes:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/tree/master/conversational-context-and-concurrency/index.md">https://github.com/nuprl/hopl-s2017/tree/master/conversational-context-and-concurrency/index.md</a></li></ul> + +<p>Discussion summary:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-01-31.md">https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017&ndash;01&ndash;31.md</a></li></ul> + + Introducing HOPL 2017 + http://prl.ccs.neu.edu/blog/2017/02/15/introducing-hopl-2017/?utm_source=HOPL&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-15-introducing-hopl-2017 + Wed, 15 Feb 2017 01:21:37 UT + Ben Greenman + +<p>This semester at Northeastern, Matthias Felleisen is organizing the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">History of Programming Languages</a> seminar. Look for posts tagged <code>HOPL</code> for updates from the lectures.</p> +<!-- more--> + +<p>Once every 6 to 8 years (i.e., once every batch of Ph.D. students?), <a href="http://www.ccs.neu.edu/home/matthias">Matthias Felleisen</a> teaches History of Programming Languages. Nominally, the course is a seminar. But unlike a typical seminar course, weekly topics are not the technical details from a handful of papers. Rather:</p> + +<blockquote> + <p>The primary goal is to understand (some of) the discipline as it exists today and how some of its major themes evolved.</p></blockquote> + +<blockquote> + <p>The secondary goal is to develop basic skills for understanding and describing research themes. Every student will learn to study a theme via a series of papers, prepare an annotated bibliography, and present the key steps in the evolution of the theme.</p></blockquote> + +<p><strong>Themes</strong> is the operative word. To set the tone, this semester started with &ldquo;themes that NUPRL faculty members have developed over the many decades of their careers.&rdquo;</p> + +<ul> + <li>Matthias, <em>Full Abstraction: From PCF to SPCF</em></li> + <li>Jan Vitek, <em>From Encapsulation to Ownership</em></li> + <li>Will Clinger, <em>Garbage Collection vs. Manual Allocation</em></li> + <li>Olin Shivers, <em>Higher-order Flow Analysis</em></li> + <li>Amal Ahmed, <em>Logical Relations: Stepping Beyond Toy Languages</em></li> + <li>Matthias, <em>Programming Languages and Calculi</em></li> + <li>Jan-Willem van de Meent, <em>Rescoring Strategies for Probabilistic Programs</em></li> + <li>(upcoming) Mitch Wand, <em>Analysis-Based Program Transformation</em></li> + <li>(upcoming) Frank Tip, <em>Refactoring</em></li></ul> + +<p>At this point in the course, we are just starting with the student presentations. As these presentations happen, we plan to push updates to this blog. All presentation materials are in the course repository:</p> + +<ul> + <li><a href="https://github.com/nuprl/hopl-s2017">https://github.com/nuprl/hopl-s2017</a></li></ul> + +<p>Speakers&rsquo; notes and annotated bibliographies are in top-level folders in the repo. Discussion summaries and &ldquo;unofficial&rdquo; notes are in the top-level <a href="https://github.com/nuprl/hopl-s2017/tree/master/lecture_notes"><code>lecture_notes/</code></a> folder.</p> + +<p>The list of upcoming presentations is online (along with <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/Summary___Materials.html">the papers</a> each presentation is based on):</p> + +<ul> + <li><a href="http://www.ccs.neu.edu/home/matthias/7480-s17/lectures.html">http://www.ccs.neu.edu/home/matthias/7480-s17/lectures.html</a></li></ul> + +<p>Blogs posts for each talk should appear 2 weeks after the talk happens.</p> + +<hr /> + +<p>Links to past editions of HOPL:</p> + +<ul> + <li><a href="http://www.ccs.neu.edu/home/matthias/369-s10/index.html">Spring 2010</a></li> + <li><a href="http://www.ccs.neu.edu/home/matthias/369-s04/index.html">Spring 2004</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/ICFP.atom.xml b/blog/feeds/ICFP.atom.xml new file mode 100644 index 00000000..31a45043 --- /dev/null +++ b/blog/feeds/ICFP.atom.xml @@ -0,0 +1,41 @@ + + + PRL Blog: Posts tagged 'ICFP' + + + urn:http-prl-ccs-neu-edu:-blog-tags-ICFP-html + 2016-06-07T11:53:47Z + + ICFP 2016: looking for student volunteers + + urn:http-prl-ccs-neu-edu:-blog-2016-06-07-icfp-2016-looking-for-student-volunteers + 2016-06-07T11:53:47Z + 2016-06-07T11:53:47Z + + Gabriel Scherer + +<p>If you are a student, you should consider <a href="http://goo.gl/forms/fKg3vpjNryBlGWB32">applying</a> to become an ICFP 2016 student volunteer! The deadline for application is July 31st, 2016.</p> +<!-- more--> + +<p><a href="http://conf.researchr.org/attending/icfp-2016/Student+Volunteers">ICFP 2016</a>, the Internal Conference on Functional Programming, is happening in Nara, Japan. If you are a student, you may be interest in being a Student Volunteer: you help run the conference, and in exchange do not pay registration fees &mdash; but you still have to find funding for the travel, hosting, and dinners. Quoting the <a href="http://conf.researchr.org/attending/icfp-2016/Student+Volunteers">Student Volunteer</a> webpage:</p> + +<blockquote> + <p>ICFP is pleased to offer a number of opportunities for student volunteers, who are vital to the efficient operation and continued success of the conference each year. The student volunteer program is a chance for students from around the world to participate in the conferences whilst assisting us in preparing and running the event.</p> + <p>Job assignments for student volunteers include assisting with technical sessions, workshops, tutorials and panels, helping the registration desk, operating the information desk, helping with traffic flow, and general assistance to keep the conferences running smoothly.</p> + <p>The Student Volunteer Program helps more students attend the ICFP conference by covering conference fees (but not travel or lodging expenses) in exchange for a fixed number of work hours (usually from 8 to 12) helping with the conference organization (registration and information desks, assistance during talk sessions, etc.).</p> + <p>The Student Volunteer registration covers:</p> + <ul> + <li>Access to all workshops and the main conference,</li> + <li>Daily lunches and coffee breaks,</li> + <li>Access to social events, including the banquet.</li></ul> + <p>To apply, please fill the <a href="http://goo.gl/forms/fKg3vpjNryBlGWB32">following form</a>.</p> + <p>The application deadline is July 31st, 2016. Applications after this date may be considered pending availability.</p> + <p>You can send questions about student volunteering to <code>icfp-SV at researchr dot org</code>.</p></blockquote> + +<p>I was &ldquo;student volunteer captain&rdquo; at ICFP last year in Vancouver, and I will do it again this year. My entirely personal take on the thing is that being a Student Volunteer is worth it, but that being a Captain is too much work.</p> + +<p>The main downside of being a volunteer is some of the shifts are at the registration desk, and they may imply missing some of the talks &mdash; and also you may have to get up early for your duties. The upsides are many. You get belong to a small group of nice people. You have interactions with many people without much effort; you will enjoy the sparks of gratitude in the eyes of the &ldquo;Where is Workshop Room B2?&rdquo; crowd. You have a small peek into the kind of work involved in running a conference; most people actually working on the organization (we SVs are hobbyists) are pouring surprising amount of work. Also, you learn to fold tee-shirts very well, if you&rsquo;re on &ldquo;bag stuffing&rdquo; duty.</p> + +<p>Being a student volunteer can be combined with other forms of travel support, such as SIGPLAN PAC funding; see the <a href="http://conf.researchr.org/attending/icfp-2016/student-travel-support">travel support page</a> for more details.</p> + +<p>Another thing you should think about is applying to <a href="http://conf.researchr.org/track/icfp-2016/PLMW-ICFP-2016">PLMW</a>, the Programming Languages Mentoring Workshop that happens at ICFP, POPL, and PLDI. PLMW funding covers the whole conference cost (travel, housing, registration, dinners), so if you get PLMW funding you have no financial motivation to be a student volunteer. This year, PLMW focuses on early career graduate students.</p> \ No newline at end of file diff --git a/blog/feeds/ICFP.rss.xml b/blog/feeds/ICFP.rss.xml new file mode 100644 index 00000000..0ebb349c --- /dev/null +++ b/blog/feeds/ICFP.rss.xml @@ -0,0 +1,41 @@ + + + + PRL Blog: Posts tagged 'ICFP' + PRL Blog: Posts tagged 'ICFP' + http://prl.ccs.neu.edu/blog/tags/ICFP.html + Tue, 07 Jun 2016 11:53:47 UT + Tue, 07 Jun 2016 11:53:47 UT + 1800 + + ICFP 2016: looking for student volunteers + http://prl.ccs.neu.edu/blog/2016/06/07/icfp-2016-looking-for-student-volunteers/?utm_source=ICFP&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-07-icfp-2016-looking-for-student-volunteers + Tue, 07 Jun 2016 11:53:47 UT + Gabriel Scherer + +<p>If you are a student, you should consider <a href="http://goo.gl/forms/fKg3vpjNryBlGWB32">applying</a> to become an ICFP 2016 student volunteer! The deadline for application is July 31st, 2016.</p> +<!-- more--> + +<p><a href="http://conf.researchr.org/attending/icfp-2016/Student+Volunteers">ICFP 2016</a>, the Internal Conference on Functional Programming, is happening in Nara, Japan. If you are a student, you may be interest in being a Student Volunteer: you help run the conference, and in exchange do not pay registration fees &mdash; but you still have to find funding for the travel, hosting, and dinners. Quoting the <a href="http://conf.researchr.org/attending/icfp-2016/Student+Volunteers">Student Volunteer</a> webpage:</p> + +<blockquote> + <p>ICFP is pleased to offer a number of opportunities for student volunteers, who are vital to the efficient operation and continued success of the conference each year. The student volunteer program is a chance for students from around the world to participate in the conferences whilst assisting us in preparing and running the event.</p> + <p>Job assignments for student volunteers include assisting with technical sessions, workshops, tutorials and panels, helping the registration desk, operating the information desk, helping with traffic flow, and general assistance to keep the conferences running smoothly.</p> + <p>The Student Volunteer Program helps more students attend the ICFP conference by covering conference fees (but not travel or lodging expenses) in exchange for a fixed number of work hours (usually from 8 to 12) helping with the conference organization (registration and information desks, assistance during talk sessions, etc.).</p> + <p>The Student Volunteer registration covers:</p> + <ul> + <li>Access to all workshops and the main conference,</li> + <li>Daily lunches and coffee breaks,</li> + <li>Access to social events, including the banquet.</li></ul> + <p>To apply, please fill the <a href="http://goo.gl/forms/fKg3vpjNryBlGWB32">following form</a>.</p> + <p>The application deadline is July 31st, 2016. Applications after this date may be considered pending availability.</p> + <p>You can send questions about student volunteering to <code>icfp-SV at researchr dot org</code>.</p></blockquote> + +<p>I was &ldquo;student volunteer captain&rdquo; at ICFP last year in Vancouver, and I will do it again this year. My entirely personal take on the thing is that being a Student Volunteer is worth it, but that being a Captain is too much work.</p> + +<p>The main downside of being a volunteer is some of the shifts are at the registration desk, and they may imply missing some of the talks &mdash; and also you may have to get up early for your duties. The upsides are many. You get belong to a small group of nice people. You have interactions with many people without much effort; you will enjoy the sparks of gratitude in the eyes of the &ldquo;Where is Workshop Room B2?&rdquo; crowd. You have a small peek into the kind of work involved in running a conference; most people actually working on the organization (we SVs are hobbyists) are pouring surprising amount of work. Also, you learn to fold tee-shirts very well, if you&rsquo;re on &ldquo;bag stuffing&rdquo; duty.</p> + +<p>Being a student volunteer can be combined with other forms of travel support, such as SIGPLAN PAC funding; see the <a href="http://conf.researchr.org/attending/icfp-2016/student-travel-support">travel support page</a> for more details.</p> + +<p>Another thing you should think about is applying to <a href="http://conf.researchr.org/track/icfp-2016/PLMW-ICFP-2016">PLMW</a>, the Programming Languages Mentoring Workshop that happens at ICFP, POPL, and PLDI. PLMW funding covers the whole conference cost (travel, housing, registration, dinners), so if you get PLMW funding you have no financial motivation to be a student volunteer. This year, PLMW focuses on early career graduate students.</p> \ No newline at end of file diff --git a/blog/feeds/NEPLS.atom.xml b/blog/feeds/NEPLS.atom.xml new file mode 100644 index 00000000..70ea7bfa --- /dev/null +++ b/blog/feeds/NEPLS.atom.xml @@ -0,0 +1,69 @@ + + + PRL Blog: Posts tagged 'NEPLS' + + + urn:http-prl-ccs-neu-edu:-blog-tags-NEPLS-html + 2016-09-15T21:18:45Z + + NEPLS on October 7th at Northeastern University + + urn:http-prl-ccs-neu-edu:-blog-2016-09-15-nepls-on-october-7th-at-northeastern-university + 2016-09-15T21:18:45Z + 2016-09-15T21:18:45Z + + Ben Greenman + +<p>The next edition of the New England Programming Language Seminar (NEPLS) will be held on Friday, October 7th at Northeastern University. Organizers are Gabriel Scherer and Max New. See you there!</p> +<!-- more--> + +<p>Here is the official announcement from the NEPLS mailing list.</p> + +<blockquote> + <p>Hi everyone,</p> + <p>The next New England Programming Languages and Systems Symposium will take place on</p> + <blockquote> + <p> Friday, October 7th 2016</p></blockquote> + <p>at</p> + <blockquote> + <p> Northeastern University, Boston.</p></blockquote> + <p>Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is</p> + <blockquote> + <p> Sunday, September 25th.</p></blockquote> + <p>Acceptance notifications will be out on Thursday, September 28th.</p> + <p>Send your proposal to</p> + <blockquote> + <p> <a href="mailto:nepls-talks-committee@lists.cs.brown.edu">nepls-talks-committee@lists.cs.brown.edu</a></p></blockquote> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <blockquote> + <p> <a href="http://www.nepls.org/">http://www.nepls.org/</a></p></blockquote></blockquote> + +<p>To subscribe to the NEPLS mailing list, visit this page:</p> + +<p><a href="https://lists.cs.brown.edu/sympa/subscribe/nepls">https://lists.cs.brown.edu/sympa/subscribe/nepls</a></p> + + NEPLS on May 31st at UMass, Amherst + + urn:http-prl-ccs-neu-edu:-blog-2016-05-03-nepls-on-may-31st-at-umass-amherst + 2016-05-03T08:21:07Z + 2016-05-03T08:21:07Z + + Gabriel Scherer + +<p>It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!</p> +<!-- more--> + +<blockquote> + <p>The next New England Programming Languages and Systems Symposium will take place on Tuesday, May 31st 2016 at University of Massachusetts, Amherst. Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is Tuesday, May 17th. Send your proposal to talks@nepls.org.</p> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <p> http://www.nepls.org/</p></blockquote> + +<p>I&rsquo;m personally fond of such regional events, which is a great time to learn about the research around us in a less formal and exhausting setting than a 200-attendees conference.</p> + +<p>If you are in the area, please consider applying to talk about your work. If one of your colleague is working on something you find exciting, please invite them to apply!</p> \ No newline at end of file diff --git a/blog/feeds/NEPLS.rss.xml b/blog/feeds/NEPLS.rss.xml new file mode 100644 index 00000000..b5108637 --- /dev/null +++ b/blog/feeds/NEPLS.rss.xml @@ -0,0 +1,67 @@ + + + + PRL Blog: Posts tagged 'NEPLS' + PRL Blog: Posts tagged 'NEPLS' + http://prl.ccs.neu.edu/blog/tags/NEPLS.html + Thu, 15 Sep 2016 21:18:45 UT + Thu, 15 Sep 2016 21:18:45 UT + 1800 + + NEPLS on October 7th at Northeastern University + http://prl.ccs.neu.edu/blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/?utm_source=NEPLS&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-09-15-nepls-on-october-7th-at-northeastern-university + Thu, 15 Sep 2016 21:18:45 UT + Ben Greenman + +<p>The next edition of the New England Programming Language Seminar (NEPLS) will be held on Friday, October 7th at Northeastern University. Organizers are Gabriel Scherer and Max New. See you there!</p> +<!-- more--> + +<p>Here is the official announcement from the NEPLS mailing list.</p> + +<blockquote> + <p>Hi everyone,</p> + <p>The next New England Programming Languages and Systems Symposium will take place on</p> + <blockquote> + <p> Friday, October 7th 2016</p></blockquote> + <p>at</p> + <blockquote> + <p> Northeastern University, Boston.</p></blockquote> + <p>Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is</p> + <blockquote> + <p> Sunday, September 25th.</p></blockquote> + <p>Acceptance notifications will be out on Thursday, September 28th.</p> + <p>Send your proposal to</p> + <blockquote> + <p> <a href="mailto:nepls-talks-committee@lists.cs.brown.edu">nepls-talks-committee@lists.cs.brown.edu</a></p></blockquote> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <blockquote> + <p> <a href="http://www.nepls.org/">http://www.nepls.org/</a></p></blockquote></blockquote> + +<p>To subscribe to the NEPLS mailing list, visit this page:</p> + +<p><a href="https://lists.cs.brown.edu/sympa/subscribe/nepls">https://lists.cs.brown.edu/sympa/subscribe/nepls</a></p> + + NEPLS on May 31st at UMass, Amherst + http://prl.ccs.neu.edu/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/?utm_source=NEPLS&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-03-nepls-on-may-31st-at-umass-amherst + Tue, 03 May 2016 08:21:07 UT + Gabriel Scherer + +<p>It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!</p> +<!-- more--> + +<blockquote> + <p>The next New England Programming Languages and Systems Symposium will take place on Tuesday, May 31st 2016 at University of Massachusetts, Amherst. Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is Tuesday, May 17th. Send your proposal to talks@nepls.org.</p> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <p> http://www.nepls.org/</p></blockquote> + +<p>I&rsquo;m personally fond of such regional events, which is a great time to learn about the research around us in a less formal and exhausting setting than a 200-attendees conference.</p> + +<p>If you are in the area, please consider applying to talk about your work. If one of your colleague is working on something you find exciting, please invite them to apply!</p> \ No newline at end of file diff --git a/blog/feeds/PI-meeting.atom.xml b/blog/feeds/PI-meeting.atom.xml new file mode 100644 index 00000000..de54e047 --- /dev/null +++ b/blog/feeds/PI-meeting.atom.xml @@ -0,0 +1,154 @@ + + + PRL Blog: Posts tagged 'PI meeting' + + + urn:http-prl-ccs-neu-edu:-blog-tags-PI-meeting-html + 2017-08-22T15:54:06Z + + Gradual Typing Across the Spectrum, part II + + urn:http-prl-ccs-neu-edu:-blog-2017-08-22-gradual-typing-across-the-spectrum-part-ii + 2017-08-22T15:54:06Z + 2017-08-22T15:54:06Z + + Ben Greenman + +<p>Last week, Northeastern hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:</p> + +<p><a href="http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html">http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html</a></p> + +<p>A common thread among the talks was the question: <em>how to convert a research idea into a tool for software developers?</em></p> +<!-- more--> + +<p>In my mind, gradual typing <em>is</em> an answer to one instance of this question. The research idea is strong static type systems, and the software developers are the millions using dynamically typed languages. I know that static typing can make programs easier to write and maintain. The developers know that dynamic typing has benefits; moreover they know better than to migrate their code from one language to another on a whim. Gradual typing is a linguistic solution to the problem of <em>adding</em> the benefits of static typing to a dynamically typed language.</p> + +<p>Enough opinions, let&rsquo;s talk about the talks.</p> + +<p>The morning session consisted of four talks:</p> + +<ul> + <li> + <p><a href="https://www.cs.umd.edu/people/milod">Milod Kazerounian</a> (<a href="https://www.cs.umd.edu/">UMD</a>) spoke about upgrading the <a href="https://github.com/plum-umd/rdl">RDL</a> type checker for Ruby with support for refinement types. The idea is to compile Ruby code and types to <a href="https://emina.github.io/rosette/">Rosette</a>, and profit from <a href="http://yices.csl.sri.com/papers/cav2007.pdf">SMT</a>-assisted type checking.</p></li> + <li> + <p><a href="http://ambrosebs.com/">Ambrose Bonnaire-Sergeant</a> (<a href="https://www.cs.indiana.edu/">IU</a>, <a href="http://ambrosebs.com/talks/squash-work-boston-pi-2017.pdf">slides</a>) has been inferring <em>useful</em> <a href="http://typedclojure.org/">Typed Clojure</a> types through dynamic analysis of Clojure programs. His tool observes how values flow through a program at run-time, then lifts these observations into possibly-recursive, possibly-incorrect type annotations. The surprising result is that the tool quickly (1&ndash;2 seconds per unit test, I think) infers types that can help a developer start annotating a program.</p></li> + <li> + <p><a href="http://ccs.neu.edu/~types/">Ben Greenman</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>, <a href="http://homedirs.ccs.neu.edu/types/resources/talks/preservation-types.pdf">slides</a>) explained why he is implementing a semantics for <a href="https://github.com/racket/typed-racket">Typed Racket</a> inspired by Michael Vitousek&rsquo;s work on <a href="http://homes.soic.indiana.edu/mvitouse/papers/popl17.pdf">Reticulated Python</a>. The &ldquo;why&rdquo; is &ldquo;performance&rdquo;. The Reticulated semantics will enforce a notion of tag soundness in kind of <a href="https://en.wikipedia.org/wiki/Deal_with_the_Devil">devils contract</a> to improve performance.</p></li> + <li> + <p><a href="https://cs.brown.edu/~ptunnell/">Preston Tunnell-Wilson</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tpk-crowdsource-lang-design/">ONWARD 2017</a>) recently sent questions about programming language design to <a href="https://www.mturk.com/mturk/welcome">Mechanical Turk</a> workers. Survey says, developers have extremely diverse opinions about what they <em>expect</em> and what they <em>want</em> regarding scope, inheritance, and infix operators.</p></li></ul> + +<p>In the early afternoon, we had two talks on similar themes as the morning session:</p> + +<ul> + <li> + <p><a href="https://github.com/akuhlens">Andre Kuhlenschmidt</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is exploring the design space of efficient implementations for run-time type checks. The main challenge is how to <em>monitor</em> higher-order data in a way that efficiently performs type checks and can help the programmer debug any failed checks. This talk presented data comparing two approaches to the program; I believe the latter, improved approach is based on <a href="http://homepages.inf.ed.ac.uk/wadler/papers/coercions/coercions.pdf">coercions</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) explained that there are many ways to adapt type soundness to a gradually typed language, and presented some data comparing Typed Racket&rsquo;s <em>generalized soudness</em> to Reticulated Python&rsquo;s <em>tag soundness</em>. The data suggests that tag soundness never adds an order-of-magnitude slowdown.</p></li></ul> + +<p>Next on the schedule were two talks about implementing advanced type systems in Racket&rsquo;s macro expander (think: meta-level linguistic re-use, capture-avoiding substitution for free!)</p> + +<ul> + <li> + <p><a href="https://github.com/iitalics">Milo Turner</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) first showed how to implement <a href="https://gankro.github.io/blah/linear-rust/#definitions-and-the-state-of-rust">linear and affine</a> type systems using <a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html">syntax-parse</a>, and second presented a simpler implementation using the <a href="http://docs.racket-lang.org/turnstile/index.html">Turnstile</a> library.</p></li> + <li> + <p><a href="http://www.davidchristiansen.dk/">David Christiansen</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is building <a href="https://github.com/david-christiansen/pudding">a proof assistant</a> in Racket. This talk focused on the design and implementation of proof tactics.</p></li></ul> + +<p>After a short break, we heard about something completely different:</p> + +<ul> + <li><a href="http://justinpombrio.net/">Justin Pombrio</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/research/plt/dl/icfp2017/">ICFP 2017</a>) taught us to interpet the scoping rules of a &ldquo;core&rdquo; language as a preorder. Using the preorder, he then showed how to <em>infer</em> the scoping rules of any &ldquo;surface&rdquo; language based on its translation to the &ldquo;core&rdquo;.</li></ul> + +<p>Last summer and fall, Jeremy Siek hosted two REUs (<a href="https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5517&amp;from=fund">research experience for undergraduates</a>) at Indiana University. The two students gave the next talks:</p> + +<ul> + <li> + <p>Di Zhong (<a href="https://www.cs.indiana.edu/">IU</a>) talked about implementing interpreters in Racket, Python, and Haskell. As I understand, this was a hands-on experience through <a href="https://www.cis.upenn.edu/~bcpierce/tapl/">TAPL</a> and <a href="https://redex.racket-lang.org/">the Redex book</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="https://www.cs.indiana.edu/">IU</a>) demonstrated her implementation of <a href="http://theory.stanford.edu/~aiken/publications/papers/popl94.pdf">conditional types</a> for <a href="https://github.com/mvitousek/reticulated">Reticulated</a>.</p></li></ul> + +<p>Finally,</p> + +<ul> + <li><a href="https://nikivazou.github.io/">Niki Vazou</a> (<a href="https://www.cs.umd.edu/">UMD</a>) presented a theory of gradual refinement types. Any &ldquo;holes&rdquo; in the refinements introduce a search problem; type checking attempts to solve the problem by finding a predicate that unifies a function definition and its callers.</li></ul> + +<p>This meeting was a great opportunity to reflect on the recent past and share opinions on what&rsquo;s worth pursuing in the future. Many thanks to the participants, and to the NSF for the support!</p> + +<blockquote> + <p>If you want to know about the future, you need to ask the young people who will create it. Young people don&rsquo;t know what can&rsquo;t be done, and so they go ahead and do it. &mdash; <a href="https://www.youtube.com/watch?v=sM1bNR4DmhU">Ivan Sutherland</a></p></blockquote> + + Gradual Typing Across the Spectrum + + urn:http-prl-ccs-neu-edu:-blog-2016-05-18-gradual-typing-across-the-spectrum + 2016-05-18T07:58:56Z + 2016-05-18T07:58:56Z + + Asumu Takikawa + +<blockquote> + <p>Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. &mdash; Matthias Felleisen</p></blockquote> + +<p>Yesterday we hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found <a href="http://prl.ccs.neu.edu/gtp/pi2016/pi2016.html">here</a>). In case you aren&rsquo;t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.</p> + +<p>Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing&mdash;the ideas that cross language and platform barriers.</p> +<!-- more--> + +<p>There were a good number of both institutions and programming languages represented at the meeting, with researchers from all of <a href="http://cs.brown.edu/research/plt/">Brown University</a>, <a href="https://wonks.github.io/">Indiana University</a>, <a href="http://prl.ccs.neu.edu/">Northeastern University</a>, and the <a href="http://www.cs.umd.edu/projects/PL/">University of Maryland</a>. The languages that we work on cover a broad subset of the dynamically-typed languages: Clojure, JavaScript, R, Racket, Ruby, Pyret, and Python.</p> + +<div class="figure"><img src="/img/2016-day-slide-4.png" alt="" /> + <p class="caption"></p></div> + +<p>The specific research artifacts that were represented include <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>, <a href="https://github.com/plum-umd/rdl">RDL</a> (contracts for Ruby), <a href="http://plg.uwaterloo.ca/~dynjs/strongscript/">StrongScript</a>, <a href="http://typedclojure.org/">Typed Clojure</a>, and <a href="http://docs.racket-lang.org/ts-guide/index.html">Typed Racket</a>.</p> + +<p>In this blog post, I&rsquo;ll summarize some of the key research themes that were brought up at the meeting. Since I can&rsquo;t go into too much detail about every topic, I will link to the relevant research papers and other resources.</p> + +<p>At a high level, the talks covered four major facets of gradual typing: expressiveness, performance, usability, and implementation techniques.</p> + +<h2 id="expressiveness">Expressiveness</h2> + +<p>By expressiveness, I mean what kinds of language features a gradual type system supports and the richness of the reasoning that the type system provides. Since gradual typing is about augmenting existing dynamically-typed languages, a gradual type system should support the language features that programmers actually use.</p> + +<p>This is why recent implementations of gradual typing have focused on enabling object-oriented programming, since objects are widely used in nearly all dynamically-typed languages in use today. Unfortunately, since different languages have wildly different object systems, it&rsquo;s hard to compare research on gradual OO languages. Ben Chung is working to address this by coming up with a formal model that tries to unify various accounts of objects in order to better explain the design tradeoffs. His goal is to cover the core ideas in Reticulated Python, StrongScript, and Typed Racket.</p> + +<p>Of course, dynamically-typed languages have a lot more than OO features. Along these lines, I gave a talk on how at NU we&rsquo;re working to extend Typed Racket to cover everything from objects (my thesis topic), first-class modules (Dan Feltey&rsquo;s MS project), and higher-order contracts (Brian LaChance&rsquo;s MS project).</p> + +<p>On the other side, as programs get more complex, programmers may wish to write richer type specifications that provide even more guarantees. This makes gradual typing a wide spectrum that goes from completely untyped, fully typed, and then beyond to dependently typed. Andrew Kent and David Christiansen both presented work that takes gradual typing beyond ordinary typed reasoning with dependent types.</p> + +<p>Andrew presented an extension of Typed Racket that adds type refinements that can check rich properties (like vector bounds) that are found in real Racket code (see his RacketCon <a href="https://www.youtube.com/watch?v=ejFJIAsvdEg">talk</a> and recent <a href="http://arxiv.org/pdf/1511.07033.pdf">PLDI paper</a>). David Christiansen followed with a talk about adding dependent type theory to Typed Racket, which would allow correct-by-construction programming using a Nuprl-like proof system (he had a very cool GUI proof assistant demo in his slides!).</p> + +<h2 id="performance">Performance</h2> + +<div class="figure"><img src="/img/2016-day-slide-8.png" alt="" /> + <p class="caption"></p></div> + +<p>One of the key practical concerns about gradual typing is its performance overhead. It&rsquo;s a concern because in order to ensure type safety, a gradually-typed language implementation needs to install dynamic checks between the typed and untyped parts of a program. This catches any inconsistencies between the typed interfaces and how the untyped code may call into them.</p> + +<p>Ben Greenman gave an upbeat talk that set the stage for this topic, pointing out some key lessons that we&rsquo;ve learned about performance from building Typed Racket. The main idea he presented (also the topic of our <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016 paper</a>) is that to evaluate a gradual type system, you want to explore different ways of adding types to a program and see how much it costs. This evaluation effort started with Typed Racket, but he and Zeina Migeed are working on expanding it to Reticulated Python.</p> + +<p>From IU, Andre Kuhlenschmidt and Deyaaeldeen Almahallawi are exploring how ahead-of-time (AOT) compilation strategies could help reduce the cost of gradual typing. In particular, they are working on implementing <a href="https://github.com/deyaaeldeen/Schml">Schml</a>: a compiler from the gradually-typed lambda calculus to C.</p> + +<p>In addition to AOT compilation, the folks at IU are exploring tracing JIT compilation as a means to make gradual typing faster. More specifically, Spenser Bauman talked about Pycket, an alternative implementation of Racket that uses RPython/PyPy to dramatically lower the overhead of gradual typing (also see the <a href="https://www.youtube.com/watch?v=GOfIY8NHAqg">recording</a> of Spenser&rsquo;s talk on the topic at RacketCon and his <a href="http://homes.soic.indiana.edu/samth/pycket-draft.pdf">ICFP paper</a>).</p> + +<h2 id="usability">Usability</h2> + +<p>On the usability side, both Shriram Krishnamurthi and Ambrose Bonnaire-Sergeant made observations on what it takes to get gradual typing in the hands of real software developers.</p> + +<p>Shriram approached the topic from the angle of CS education, which is the focus of the <a href="http://www.pyret.org">Pyret</a> language, and shared what the Brown language group is working on. While Pyret doesn&rsquo;t exactly fit the mold of gradual typing, it&rsquo;s a close cousin since it&rsquo;s a dynamically-typed language that explicitly takes design cues from the best parts of statically-typed languages. That approach lets CS beginners think in terms of types (the approach spearheaded by <a href="http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html">HtDP</a> and <a href="http://www.bootstrapworld.org/">Bootstrap</a>) without having to battle a typechecker from the start.</p> + +<p>For professional software developers, a major concern with gradual typing is that writing type annotations may be a tedious and time intensive task. Ambrose, who is the creator of Typed Clojure, shared some preliminary work on how to cut down on the tedium by inferring gradual type annotations by instrumenting programs for a dynamic analysis. The hope is to be able to infer both recursive and polymorphic type annotations automatically from tests (you may also be interested in Ambrose&rsquo;s recent <a href="http://frenchy64.github.io/papers/esop16-short.pdf">ESOP paper</a> on Typed Clojure).</p> + +<h2 id="implementation-techniques">Implementation Techniques</h2> + +<p>Finally, several talks focused on alternative implementation techniques for gradual typing that provide a variety of software engineering benefits for implementers.</p> + +<p>From Maryland, Brianna Ren gave a talk on Hummingbird, a just-in-time typechecker for Ruby programs (also see the upcoming <a href="http://www.cs.umd.edu/~jfoster/papers/pldi16.pdf">PLDI paper</a> by Brianna and Jeff Foster). The basic idea is that it&rsquo;s hard to implement a traditional static typechecker for a language that heavily relies on metaprogramming, in which the fields/methods of classes may be rewritten at run-time. This is particularly tricky for frameworks like Ruby on Rails. Instead of checking types at compile-time, Hummingbird actually executes the typechecker at run-time in order to be able to accurately check programs that use run-time metaprogramming. To reduce overheads, she uses a cache for typechecking that is invalidated when classes are modified.</p> + +<p>Stephen Chang gave a very different view on metaprogramming in his talk, which focused on <em>implementing</em> typecheckers using metaprogramming (the <a href="http://docs.racket-lang.org/trivial/index.html">trivial</a> Typed Racket library is an offshoot of this work). His key idea is that typecheckers share many aspects with macro-based metaprogramming systems, such as the need to traverse syntax and annotate it with information. Since they share so much in common, why not just implement the typechecker as a macro? Stephen demonstrates that not only is this possible, but that it&rsquo;s possible to implement a wide variety of type system features this way including (local) type inference. The connection to gradual typing is that even a gradual type system can be implemented as a metaprogram by integrating the generation of dynamic checks into the macro transformation process.</p> + +<p>The last talk of the day (but certainly not the least), was by Michael Vitousek, who focused on the <em>transient</em> implementation of gradual typing (first described in his <a href="http://homes.soic.indiana.edu/mvitouse/papers/dls14.pdf">DLS paper</a>). Traditionally, gradual type systems have implemented their dynamic checks using <a href="https://en.wikipedia.org/wiki/Proxy_pattern">proxy</a> objects that wrap method implementations with both pre- and post-checks. Unfortunately, this implementation technique often conflicts with the underlying language. Since proxying changes the identity of an object it can interfere with object equality tests. Instead, the transient approach bakes the dynamic checks into and throughout the typed code to implement a &ldquo;defense in depth&rdquo; against inconsistencies with untyped code. The great thing about this implementation technique is that it doesn&rsquo;t demand any specialized support from the underlying language runtime and is therefore easy to port to other languages (like JavaScript).</p> + +<h2 id="conclusion">Conclusion</h2> + +<div class="figure"><img src="/img/2016-day-slide-3.png" alt="" /> + <p class="caption"></p></div> + +<p>Hopefully this blog post helps provide a better picture of the state of gradual typing research. The exciting thing about gradual typing is that it contains both interesting theoretical problems and also connects to the practical needs of software developers.</p> \ No newline at end of file diff --git a/blog/feeds/PI-meeting.rss.xml b/blog/feeds/PI-meeting.rss.xml new file mode 100644 index 00000000..7ff9dc46 --- /dev/null +++ b/blog/feeds/PI-meeting.rss.xml @@ -0,0 +1,152 @@ + + + + PRL Blog: Posts tagged 'PI meeting' + PRL Blog: Posts tagged 'PI meeting' + http://prl.ccs.neu.edu/blog/tags/PI-meeting.html + Tue, 22 Aug 2017 15:54:06 UT + Tue, 22 Aug 2017 15:54:06 UT + 1800 + + Gradual Typing Across the Spectrum, part II + http://prl.ccs.neu.edu/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/?utm_source=PI-meeting&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-22-gradual-typing-across-the-spectrum-part-ii + Tue, 22 Aug 2017 15:54:06 UT + Ben Greenman + +<p>Last week, Northeastern hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:</p> + +<p><a href="http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html">http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html</a></p> + +<p>A common thread among the talks was the question: <em>how to convert a research idea into a tool for software developers?</em></p> +<!-- more--> + +<p>In my mind, gradual typing <em>is</em> an answer to one instance of this question. The research idea is strong static type systems, and the software developers are the millions using dynamically typed languages. I know that static typing can make programs easier to write and maintain. The developers know that dynamic typing has benefits; moreover they know better than to migrate their code from one language to another on a whim. Gradual typing is a linguistic solution to the problem of <em>adding</em> the benefits of static typing to a dynamically typed language.</p> + +<p>Enough opinions, let&rsquo;s talk about the talks.</p> + +<p>The morning session consisted of four talks:</p> + +<ul> + <li> + <p><a href="https://www.cs.umd.edu/people/milod">Milod Kazerounian</a> (<a href="https://www.cs.umd.edu/">UMD</a>) spoke about upgrading the <a href="https://github.com/plum-umd/rdl">RDL</a> type checker for Ruby with support for refinement types. The idea is to compile Ruby code and types to <a href="https://emina.github.io/rosette/">Rosette</a>, and profit from <a href="http://yices.csl.sri.com/papers/cav2007.pdf">SMT</a>-assisted type checking.</p></li> + <li> + <p><a href="http://ambrosebs.com/">Ambrose Bonnaire-Sergeant</a> (<a href="https://www.cs.indiana.edu/">IU</a>, <a href="http://ambrosebs.com/talks/squash-work-boston-pi-2017.pdf">slides</a>) has been inferring <em>useful</em> <a href="http://typedclojure.org/">Typed Clojure</a> types through dynamic analysis of Clojure programs. His tool observes how values flow through a program at run-time, then lifts these observations into possibly-recursive, possibly-incorrect type annotations. The surprising result is that the tool quickly (1&ndash;2 seconds per unit test, I think) infers types that can help a developer start annotating a program.</p></li> + <li> + <p><a href="http://ccs.neu.edu/~types/">Ben Greenman</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>, <a href="http://homedirs.ccs.neu.edu/types/resources/talks/preservation-types.pdf">slides</a>) explained why he is implementing a semantics for <a href="https://github.com/racket/typed-racket">Typed Racket</a> inspired by Michael Vitousek&rsquo;s work on <a href="http://homes.soic.indiana.edu/mvitouse/papers/popl17.pdf">Reticulated Python</a>. The &ldquo;why&rdquo; is &ldquo;performance&rdquo;. The Reticulated semantics will enforce a notion of tag soundness in kind of <a href="https://en.wikipedia.org/wiki/Deal_with_the_Devil">devils contract</a> to improve performance.</p></li> + <li> + <p><a href="https://cs.brown.edu/~ptunnell/">Preston Tunnell-Wilson</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tpk-crowdsource-lang-design/">ONWARD 2017</a>) recently sent questions about programming language design to <a href="https://www.mturk.com/mturk/welcome">Mechanical Turk</a> workers. Survey says, developers have extremely diverse opinions about what they <em>expect</em> and what they <em>want</em> regarding scope, inheritance, and infix operators.</p></li></ul> + +<p>In the early afternoon, we had two talks on similar themes as the morning session:</p> + +<ul> + <li> + <p><a href="https://github.com/akuhlens">Andre Kuhlenschmidt</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is exploring the design space of efficient implementations for run-time type checks. The main challenge is how to <em>monitor</em> higher-order data in a way that efficiently performs type checks and can help the programmer debug any failed checks. This talk presented data comparing two approaches to the program; I believe the latter, improved approach is based on <a href="http://homepages.inf.ed.ac.uk/wadler/papers/coercions/coercions.pdf">coercions</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) explained that there are many ways to adapt type soundness to a gradually typed language, and presented some data comparing Typed Racket&rsquo;s <em>generalized soudness</em> to Reticulated Python&rsquo;s <em>tag soundness</em>. The data suggests that tag soundness never adds an order-of-magnitude slowdown.</p></li></ul> + +<p>Next on the schedule were two talks about implementing advanced type systems in Racket&rsquo;s macro expander (think: meta-level linguistic re-use, capture-avoiding substitution for free!)</p> + +<ul> + <li> + <p><a href="https://github.com/iitalics">Milo Turner</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) first showed how to implement <a href="https://gankro.github.io/blah/linear-rust/#definitions-and-the-state-of-rust">linear and affine</a> type systems using <a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html">syntax-parse</a>, and second presented a simpler implementation using the <a href="http://docs.racket-lang.org/turnstile/index.html">Turnstile</a> library.</p></li> + <li> + <p><a href="http://www.davidchristiansen.dk/">David Christiansen</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is building <a href="https://github.com/david-christiansen/pudding">a proof assistant</a> in Racket. This talk focused on the design and implementation of proof tactics.</p></li></ul> + +<p>After a short break, we heard about something completely different:</p> + +<ul> + <li><a href="http://justinpombrio.net/">Justin Pombrio</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/research/plt/dl/icfp2017/">ICFP 2017</a>) taught us to interpet the scoping rules of a &ldquo;core&rdquo; language as a preorder. Using the preorder, he then showed how to <em>infer</em> the scoping rules of any &ldquo;surface&rdquo; language based on its translation to the &ldquo;core&rdquo;.</li></ul> + +<p>Last summer and fall, Jeremy Siek hosted two REUs (<a href="https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5517&amp;from=fund">research experience for undergraduates</a>) at Indiana University. The two students gave the next talks:</p> + +<ul> + <li> + <p>Di Zhong (<a href="https://www.cs.indiana.edu/">IU</a>) talked about implementing interpreters in Racket, Python, and Haskell. As I understand, this was a hands-on experience through <a href="https://www.cis.upenn.edu/~bcpierce/tapl/">TAPL</a> and <a href="https://redex.racket-lang.org/">the Redex book</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="https://www.cs.indiana.edu/">IU</a>) demonstrated her implementation of <a href="http://theory.stanford.edu/~aiken/publications/papers/popl94.pdf">conditional types</a> for <a href="https://github.com/mvitousek/reticulated">Reticulated</a>.</p></li></ul> + +<p>Finally,</p> + +<ul> + <li><a href="https://nikivazou.github.io/">Niki Vazou</a> (<a href="https://www.cs.umd.edu/">UMD</a>) presented a theory of gradual refinement types. Any &ldquo;holes&rdquo; in the refinements introduce a search problem; type checking attempts to solve the problem by finding a predicate that unifies a function definition and its callers.</li></ul> + +<p>This meeting was a great opportunity to reflect on the recent past and share opinions on what&rsquo;s worth pursuing in the future. Many thanks to the participants, and to the NSF for the support!</p> + +<blockquote> + <p>If you want to know about the future, you need to ask the young people who will create it. Young people don&rsquo;t know what can&rsquo;t be done, and so they go ahead and do it. &mdash; <a href="https://www.youtube.com/watch?v=sM1bNR4DmhU">Ivan Sutherland</a></p></blockquote> + + Gradual Typing Across the Spectrum + http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/?utm_source=PI-meeting&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-18-gradual-typing-across-the-spectrum + Wed, 18 May 2016 07:58:56 UT + Asumu Takikawa + +<blockquote> + <p>Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. &mdash; Matthias Felleisen</p></blockquote> + +<p>Yesterday we hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found <a href="http://prl.ccs.neu.edu/gtp/pi2016/pi2016.html">here</a>). In case you aren&rsquo;t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.</p> + +<p>Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing&mdash;the ideas that cross language and platform barriers.</p> +<!-- more--> + +<p>There were a good number of both institutions and programming languages represented at the meeting, with researchers from all of <a href="http://cs.brown.edu/research/plt/">Brown University</a>, <a href="https://wonks.github.io/">Indiana University</a>, <a href="http://prl.ccs.neu.edu/">Northeastern University</a>, and the <a href="http://www.cs.umd.edu/projects/PL/">University of Maryland</a>. The languages that we work on cover a broad subset of the dynamically-typed languages: Clojure, JavaScript, R, Racket, Ruby, Pyret, and Python.</p> + +<div class="figure"><img src="/img/2016-day-slide-4.png" alt="" /> + <p class="caption"></p></div> + +<p>The specific research artifacts that were represented include <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>, <a href="https://github.com/plum-umd/rdl">RDL</a> (contracts for Ruby), <a href="http://plg.uwaterloo.ca/~dynjs/strongscript/">StrongScript</a>, <a href="http://typedclojure.org/">Typed Clojure</a>, and <a href="http://docs.racket-lang.org/ts-guide/index.html">Typed Racket</a>.</p> + +<p>In this blog post, I&rsquo;ll summarize some of the key research themes that were brought up at the meeting. Since I can&rsquo;t go into too much detail about every topic, I will link to the relevant research papers and other resources.</p> + +<p>At a high level, the talks covered four major facets of gradual typing: expressiveness, performance, usability, and implementation techniques.</p> + +<h2 id="expressiveness">Expressiveness</h2> + +<p>By expressiveness, I mean what kinds of language features a gradual type system supports and the richness of the reasoning that the type system provides. Since gradual typing is about augmenting existing dynamically-typed languages, a gradual type system should support the language features that programmers actually use.</p> + +<p>This is why recent implementations of gradual typing have focused on enabling object-oriented programming, since objects are widely used in nearly all dynamically-typed languages in use today. Unfortunately, since different languages have wildly different object systems, it&rsquo;s hard to compare research on gradual OO languages. Ben Chung is working to address this by coming up with a formal model that tries to unify various accounts of objects in order to better explain the design tradeoffs. His goal is to cover the core ideas in Reticulated Python, StrongScript, and Typed Racket.</p> + +<p>Of course, dynamically-typed languages have a lot more than OO features. Along these lines, I gave a talk on how at NU we&rsquo;re working to extend Typed Racket to cover everything from objects (my thesis topic), first-class modules (Dan Feltey&rsquo;s MS project), and higher-order contracts (Brian LaChance&rsquo;s MS project).</p> + +<p>On the other side, as programs get more complex, programmers may wish to write richer type specifications that provide even more guarantees. This makes gradual typing a wide spectrum that goes from completely untyped, fully typed, and then beyond to dependently typed. Andrew Kent and David Christiansen both presented work that takes gradual typing beyond ordinary typed reasoning with dependent types.</p> + +<p>Andrew presented an extension of Typed Racket that adds type refinements that can check rich properties (like vector bounds) that are found in real Racket code (see his RacketCon <a href="https://www.youtube.com/watch?v=ejFJIAsvdEg">talk</a> and recent <a href="http://arxiv.org/pdf/1511.07033.pdf">PLDI paper</a>). David Christiansen followed with a talk about adding dependent type theory to Typed Racket, which would allow correct-by-construction programming using a Nuprl-like proof system (he had a very cool GUI proof assistant demo in his slides!).</p> + +<h2 id="performance">Performance</h2> + +<div class="figure"><img src="/img/2016-day-slide-8.png" alt="" /> + <p class="caption"></p></div> + +<p>One of the key practical concerns about gradual typing is its performance overhead. It&rsquo;s a concern because in order to ensure type safety, a gradually-typed language implementation needs to install dynamic checks between the typed and untyped parts of a program. This catches any inconsistencies between the typed interfaces and how the untyped code may call into them.</p> + +<p>Ben Greenman gave an upbeat talk that set the stage for this topic, pointing out some key lessons that we&rsquo;ve learned about performance from building Typed Racket. The main idea he presented (also the topic of our <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016 paper</a>) is that to evaluate a gradual type system, you want to explore different ways of adding types to a program and see how much it costs. This evaluation effort started with Typed Racket, but he and Zeina Migeed are working on expanding it to Reticulated Python.</p> + +<p>From IU, Andre Kuhlenschmidt and Deyaaeldeen Almahallawi are exploring how ahead-of-time (AOT) compilation strategies could help reduce the cost of gradual typing. In particular, they are working on implementing <a href="https://github.com/deyaaeldeen/Schml">Schml</a>: a compiler from the gradually-typed lambda calculus to C.</p> + +<p>In addition to AOT compilation, the folks at IU are exploring tracing JIT compilation as a means to make gradual typing faster. More specifically, Spenser Bauman talked about Pycket, an alternative implementation of Racket that uses RPython/PyPy to dramatically lower the overhead of gradual typing (also see the <a href="https://www.youtube.com/watch?v=GOfIY8NHAqg">recording</a> of Spenser&rsquo;s talk on the topic at RacketCon and his <a href="http://homes.soic.indiana.edu/samth/pycket-draft.pdf">ICFP paper</a>).</p> + +<h2 id="usability">Usability</h2> + +<p>On the usability side, both Shriram Krishnamurthi and Ambrose Bonnaire-Sergeant made observations on what it takes to get gradual typing in the hands of real software developers.</p> + +<p>Shriram approached the topic from the angle of CS education, which is the focus of the <a href="http://www.pyret.org">Pyret</a> language, and shared what the Brown language group is working on. While Pyret doesn&rsquo;t exactly fit the mold of gradual typing, it&rsquo;s a close cousin since it&rsquo;s a dynamically-typed language that explicitly takes design cues from the best parts of statically-typed languages. That approach lets CS beginners think in terms of types (the approach spearheaded by <a href="http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html">HtDP</a> and <a href="http://www.bootstrapworld.org/">Bootstrap</a>) without having to battle a typechecker from the start.</p> + +<p>For professional software developers, a major concern with gradual typing is that writing type annotations may be a tedious and time intensive task. Ambrose, who is the creator of Typed Clojure, shared some preliminary work on how to cut down on the tedium by inferring gradual type annotations by instrumenting programs for a dynamic analysis. The hope is to be able to infer both recursive and polymorphic type annotations automatically from tests (you may also be interested in Ambrose&rsquo;s recent <a href="http://frenchy64.github.io/papers/esop16-short.pdf">ESOP paper</a> on Typed Clojure).</p> + +<h2 id="implementation-techniques">Implementation Techniques</h2> + +<p>Finally, several talks focused on alternative implementation techniques for gradual typing that provide a variety of software engineering benefits for implementers.</p> + +<p>From Maryland, Brianna Ren gave a talk on Hummingbird, a just-in-time typechecker for Ruby programs (also see the upcoming <a href="http://www.cs.umd.edu/~jfoster/papers/pldi16.pdf">PLDI paper</a> by Brianna and Jeff Foster). The basic idea is that it&rsquo;s hard to implement a traditional static typechecker for a language that heavily relies on metaprogramming, in which the fields/methods of classes may be rewritten at run-time. This is particularly tricky for frameworks like Ruby on Rails. Instead of checking types at compile-time, Hummingbird actually executes the typechecker at run-time in order to be able to accurately check programs that use run-time metaprogramming. To reduce overheads, she uses a cache for typechecking that is invalidated when classes are modified.</p> + +<p>Stephen Chang gave a very different view on metaprogramming in his talk, which focused on <em>implementing</em> typecheckers using metaprogramming (the <a href="http://docs.racket-lang.org/trivial/index.html">trivial</a> Typed Racket library is an offshoot of this work). His key idea is that typecheckers share many aspects with macro-based metaprogramming systems, such as the need to traverse syntax and annotate it with information. Since they share so much in common, why not just implement the typechecker as a macro? Stephen demonstrates that not only is this possible, but that it&rsquo;s possible to implement a wide variety of type system features this way including (local) type inference. The connection to gradual typing is that even a gradual type system can be implemented as a metaprogram by integrating the generation of dynamic checks into the macro transformation process.</p> + +<p>The last talk of the day (but certainly not the least), was by Michael Vitousek, who focused on the <em>transient</em> implementation of gradual typing (first described in his <a href="http://homes.soic.indiana.edu/mvitouse/papers/dls14.pdf">DLS paper</a>). Traditionally, gradual type systems have implemented their dynamic checks using <a href="https://en.wikipedia.org/wiki/Proxy_pattern">proxy</a> objects that wrap method implementations with both pre- and post-checks. Unfortunately, this implementation technique often conflicts with the underlying language. Since proxying changes the identity of an object it can interfere with object equality tests. Instead, the transient approach bakes the dynamic checks into and throughout the typed code to implement a &ldquo;defense in depth&rdquo; against inconsistencies with untyped code. The great thing about this implementation technique is that it doesn&rsquo;t demand any specialized support from the underlying language runtime and is therefore easy to port to other languages (like JavaScript).</p> + +<h2 id="conclusion">Conclusion</h2> + +<div class="figure"><img src="/img/2016-day-slide-3.png" alt="" /> + <p class="caption"></p></div> + +<p>Hopefully this blog post helps provide a better picture of the state of gradual typing research. The exciting thing about gradual typing is that it contains both interesting theoretical problems and also connects to the practical needs of software developers.</p> \ No newline at end of file diff --git a/blog/feeds/PL-Junior.atom.xml b/blog/feeds/PL-Junior.atom.xml new file mode 100644 index 00000000..f359bb98 --- /dev/null +++ b/blog/feeds/PL-Junior.atom.xml @@ -0,0 +1,127 @@ + + + PRL Blog: Posts tagged 'PL Junior' + + + urn:http-prl-ccs-neu-edu:-blog-tags-PL-Junior-html + 2017-06-16T11:38:25Z + + Spring 2017 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + 2017-06-16T11:38:25Z + 2017-06-16T11:38:25Z + Ben Chung + Milo Davis + Ming-Ho Yee + Matt Kolosick + Dustin Jamner + Artem Pelenitsyn + Julia Belyakova + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Fall 2016 PL Junior Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + 2017-01-02T16:39:37Z + 2017-01-02T16:39:37Z + Ben Chung + Milo Davis + Ming-Ho Yee + Sam Caldwell + + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> \ No newline at end of file diff --git a/blog/feeds/PL-Junior.rss.xml b/blog/feeds/PL-Junior.rss.xml new file mode 100644 index 00000000..0ac553c7 --- /dev/null +++ b/blog/feeds/PL-Junior.rss.xml @@ -0,0 +1,113 @@ + + + + PRL Blog: Posts tagged 'PL Junior' + PRL Blog: Posts tagged 'PL Junior' + http://prl.ccs.neu.edu/blog/tags/PL-Junior.html + Fri, 16 Jun 2017 11:38:25 UT + Fri, 16 Jun 2017 11:38:25 UT + 1800 + + Spring 2017 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/?utm_source=PL-Junior&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-16-spring-2017-pl-junior-retrospective + Fri, 16 Jun 2017 11:38:25 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">PL Junior Seminar</a> is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.</p> + +<p>For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a> and Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">Early History of Smalltalk</a>. At the same time, we changed up the format from the previous semester.</p> +<!-- more--> + +<h2 id="format">Format</h2> + +<p>As discussed in <a href="http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/">last fall&rsquo;s retrospective</a>, we wanted to move from group reading and discussion towards weekly presentations. Reading a paper to prepare a presentation is quite a different experience compared to the effort that goes in when it is just for a group discussion (in our experience). With any luck, the presentation will convey some of this deeper knowledge to the rest of the group, with the result being a deep understanding on the part of the presenter and an informed, if somewhat shallower, understanding in the rest of the group. Ideally, the end result should compare favorably to simply reading the paper individually.</p> + +<p>One idea from last semester that we decided to keep is to spend a length of time (possibly an entire semester) on a topic rather than having a new topic each week. Staying on the same theme helps with retention as well as allowing for deeper investigation.</p> + +<p>In that spirit, we chose three themes for the semester: program synthesis, the sequent calculus, and logic programming. Mostly by chance, these topics have interesting connections to each other, and we even had several PL Grown-Up Seminars this semester on program synthesis!</p> + +<h2 id="synthesis">Synthesis</h2> + +<p>The first paper on program synthesis that we looked at was <a href="https://www.sri.com/sites/default/files/uploads/publications/pdf/725.pdf">A Deductive Approach to Program Synthesis</a> by Manna and Waldinger. We chose this paper because it&rsquo;s old and has a lot of citations so it&rsquo;s probably Important. It was interesting and provided an OK introduction to proof search but the method presented seems far removed from modern synthesis techniques.</p> + +<p>The next paper was <a href="https://arxiv.org/abs/1507.02988">Programmatic and Direct Manipulation, Together</a> by Chugh, Hempel, Spradlin, and Alders, which presents the <a href="https://ravichugh.github.io/sketch-n-sketch/index.html">Sketch-n-Sketch</a> system. Sketch-n-Sketch is a cool system. It demonstrates that a narrow application of synthesis - trying to fill in the constant values in a program (sketching) - can be used for great effect. We were left wondering, however, if it was too narrow an application of synthesis to give much of an indication of what the entire field is like.</p> + +<p>We concluded our program synthesis segment with <a href="http://www.cis.upenn.edu/~stevez/papers/OZ15.pdf">Type-and-Example-Directed Program Synthesis</a> by Osera and Zdancewic, another relatively recent paper. This seems like a relevant paper because we are under the impression that using examples to do synthesis is a big thing right now. Using types to constrain the search is another interesting perspective on techniques for synthesis.</p> + +<p>While each of theses papers had merits, none was so comprehensive as to be a necessary inclusion in any future look at program synthesis for pl junior</p> + +<h2 id="sequent-calculus">Sequent Calculus</h2> + +<p>We followed up the program synthesis unit with a week on the sequent calculus. The seminar presentation was based on a paper by <a href="https://hal.inria.fr/inria-00381525/document">Herbelin</a>. <a href="http://www.ccs.neu.edu/home/gasche/phd_thesis/scherer-thesis.pdf">Gabriel’s thesis</a> (chapter 4) includes maybe a more suitable modern introduction to the sequent calculus.</p> + +<p>It might have been better to do sequent calculus first because there is a modern branch of proof search based on the sequent calculus. Presenting this first would have allowed us to look into proof search for program synthesis.</p> + +<p>An additional problem is that it was insufficiently motivated. Either skipping the topic or spending more time on it would be preferable, since one week was just enough to describe the sequent calculus but not enough to apply it. For this topic to be worthwhile, it would best be used as the basis for subsequent readings that directly reference it.</p> + +<h2 id="logic-programming">Logic Programming</h2> + +<p>The topic was presented over two weeks. The first session presented/demoed Prolog as a language, and we got a sense of what logic programming could do. But it was a whirlwind tour, and we were left wondering about specific details (how proof search runs, what <code>cut</code> does).</p> + +<p>The second session presented the paper <a href="http://www.doc.ic.ac.uk/~rak/papers/kowalski-van_emden.pdf">The Semantics of Predicate Logic as a Programming Language</a>. It was interesting and insightful but left wondering how it relates to the implementation of real logic programming languages.</p> + +<p>In hindsight this was about as far as we could have gotten in just two weeks. However, complications such as the cut rule seem prevalent enough in practice that more time would be required to build up a useful understanding of logic programming</p> + +<h2 id="bonus-rounds">Bonus Rounds</h2> + +<p>We also used a few weeks to read and discuss specific papers as a group.</p> + +<p>The first paper we read was Cardelli&rsquo;s <a href="http://www.lucacardelli.name/Papers/TypefulProg.pdf">Typeful Programming</a>. We picked typeful programming because Matthias has mentioned on occasion how important he thinks it is.</p> + +<p>It was an interesting read; more of an essay than a paper. It really stood out as different from the other academic publications that we have looked at. It’s a walk through of a language design motivating each design decision in practical terms, as in things that actually help the programmer.</p> + +<p>Cardelli places great importance on polymorphism (subtyping in addition to parametric), as well as features for programming in the large such as modules and interfaces. Several features are interesting in their omission, like type inference and macros.</p> + +<p>After reading it it’s not clear why Matthias thinks it’s so important. From the perspective of modern researchers, many of the features in Cardelli&rsquo;s language seem rather mundane. However, it&rsquo;s likely that at the time he published it, these ideas were significantly newer and much less widespread.</p> + +<p>The other paper we read as a group was Alan Kay&rsquo;s <a href="http://worrydream.com/EarlyHistoryOfSmalltalk/">The Early History of Smalltalk</a>. It seems like the Smalltalk project investigated a plethora of interesting ideas about designing programming languages and environments. This article seems to confirm that but does not delve into many particulars.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>Overall this semester of pl junior went well enough that we think it makes a reasonable template for future semesters. The topics were interesting and relevant, and we mostly picked appropriate material for presentations. One downside is that we didn’t quite ‘fill out’ the semester with presentations due to scheduling and not wanting to make some people present twice. Here’s a lesson: recruit more people to the phd program (or get more undergrads) so you don’t have this problem!</p> + +<p>Having papers in a theme helped a lot over previous paper-presentation iterations of pl junior. It helped each week being able to build on what we learned last week, as opposed to having a whirlwind of unrelated topics.</p> + +<p>Writing this retrospective has also proven to be a beneficial exercise. Especially with our sequences of connected topics, looking back has allowed us to put the earlier papers into perspective and better assess both their relevance and presentation.</p> + + Fall 2016 PL Junior Retrospective + http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/?utm_source=PL-Junior&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-01-02-fall-2016-pl-junior-retrospective + Mon, 02 Jan 2017 16:39:37 UT + Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell + +<p>The <a href="http://prl.ccs.neu.edu/seminars.html">Programming Language Seminar, Junior</a> (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because</p> + +<ol> + <li>working from the <a href="https://mitpress.mit.edu/books/types-and-programming-languages">TAPL</a> presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and</li> + <li>they represent a significant increase in the reasoning power of types over programs.</li></ol> +<!-- more--> + +<p>There was a preference for learning how to implement a dependent type system, instead of spending a significant amount of time reading papers, especially dense type-theoretic papers suggested by <a href="http://purelytheoretical.com/sywtltt.html">posts</a> like <a href="http://jozefg.bitbucket.org/posts/2015-08-14-learn-tt.html">these</a>. So we followed the <a href="https://github.com/sweirich/pi-forall">pi-for-all</a> lecture series given by Stephanie Weirich at <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">OPLSS</a>, which focuses on implementing a simple dependently-typed programming language.</p> + +<p>After the pi-for-all lectures, we read chapter two of Edwin Brady’s <a href="https://eb.host.cs.st-andrews.ac.uk/writings/thesis.pdf">dissertation on implementing dependently typed languages</a>. The thesis includes a relatively approachable introduction to TT, the core dependent type theory of Epigram.</p> + +<p>Along the way, we became sidetracked by <a href="https://en.wikipedia.org/wiki/System_U#Girard.27s_paradox">Girard’s paradox</a>. In the first pi-for-all lecture, we came across the Type-in-Type rule. (In a dependent type system the term and the type languages are the same. However, we still need to distinguish what is a “program” and what is a “type,” for instance, so that we can determine that the annotation of a function’s argument is valid. So a construct in the term language is Type, which is meant to describe those things that are valid in programs where we expect to find a type). In the lecture, this prompted the comment that this (“of course”) makes our system inconsistent as a logic, but there was no further elaboration, and we could not figure out how to use this fact to show inconsistency.</p> + +<p>It turns out the reason Type-in-Type is inconsistent is quite complicated. It is explained in a <a href="https://www.cs.cmu.edu/~kw/scans/hurkens95tlca.pdf">paper</a> that we had difficulty understanding. So we turned to the students in our lab that have expertise in the area. The answer we received is that, intuitively, it is inconsistent for the same reason as Russell’s paradox (or the Burali-Forti paradox), but the actual proof is actually quite involved. The lesson we drew is that despite being “obvious,” Type-in-Type being inconsistent is not easy to prove. The way people seem to throw around this conclusion is confusing from a beginner’s point of view.</p> + +<p>The best thing about the pi-for-all series is that it demystified dependent types for us. We gained confidence in being able to whiteboard a dependent type system with the ease of System-F or STLC. If we had one complaint, the presentation of the material relied heavily on Haskell details. The unbound library to handle variables in the implementation results in a somewhat “magicy” representation of binding; it’s not clear that the benefits are so great as to outweigh the cost of just implementing alpha-equivalence and capture-avoiding-substitution. Overall they were high-quality lectures. As hinted above, we didn’t particularly care for the second lecture that was mostly a code walk-through. One advantage of watching videos was that we could speed through parts we were already comfortable with.</p> + +<p>With Edwin Brady’s dissertation, we got a glimpse of how quickly the details of a dependently typed language get hairy. Looking at you, inductive data definitions and eliminations. There were some extremely large type signatures. While this exercise boosted our confidence that we could read Serious Dependent Types™ papers, it also gave evidence that our fears of incomprehensibility were not completely unfounded.</p> + +<p>This issue appeared before in our discussion of Girard’s Paradox. In the discussion of the paradox, we got stuck when we tried to understand the very complex term that inhabited the bottom type. Dependent typing, and discussions thereof, allow very rich, meaningful, and complex types that are as complex as the code that they abstract over. While we are used to understanding these structures in code, parsing a complex type and its fundamental meaning gave us considerable difficulty.</p> + +<h2 id="thoughts-on-the-format">Thoughts on the format</h2> + +<p>Our meetings this semester were all of the form “we’ll watch this lecture or read this chapter, and then discuss it next week.” Next semester we would like to go back to presenting each week. We feel doing presentations forces the presenter to reach a deeper understanding of the material. This semester we got a relatively shallow understanding of a broad area. A deeper understanding with a narrower focus may be more beneficial (or complementary).</p> + +<p>[[Sam’s defense as Grand Convener of PL Junior: Once we picked dependent types as a topic, doing presentations was not an option. We didn’t have the expertise in the area to pick out different sub-topics and papers suitable for presentations. And, since we were short-handed (4 people each week), we would be presenting once a month!]]</p> + +<p>If we continue learning more about dependent types it would be by: 1) Doing more reading, such as the <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">ATTAPL</a> chapter or the programming in Martin-Löf’s type theory material. 2) Actually trying to implement some of the things we’ve learned this semester 3) Playing around with more of the various dependent type systems and theorem provers out there</p> + +<p>For future pl junior cohorts or anyone else in learning about dependent types: Pi-for-all is useful material, but could be condensed into two weeks (for example, by watching the lectures at 1.5x speed) instead of four. Don’t worry about Type-in-Type. The Epigram material is OK but you might be better served looking at ATTAPL first. At some point, you will have to read the dense papers, but pi-for-all is a good introduction.</p> \ No newline at end of file diff --git a/blog/feeds/PLT-Redex.atom.xml b/blog/feeds/PLT-Redex.atom.xml new file mode 100644 index 00000000..252c4980 --- /dev/null +++ b/blog/feeds/PLT-Redex.atom.xml @@ -0,0 +1,853 @@ + + + PRL Blog: Posts tagged 'PLT Redex' + + + urn:http-prl-ccs-neu-edu:-blog-tags-PLT-Redex-html + 2017-09-25T23:39:16Z + + PLT Redex FAQ + + urn:http-prl-ccs-neu-edu:-blog-2017-09-25-plt-redex-faq + 2017-09-25T23:39:16Z + 2017-09-25T23:39:16Z + Ben Greenman + Sam Caldwell + + Ben Greenman, Sam Caldwell + +<p>A short guide to Redex concepts, conventions, and common mistakes.</p> +<!-- more--> + +<hr /> + +<p><em>To contribute to this FAQ, submit issues and pull requests to: <a href="https://github.com/nuprl/website/">https://github.com/nuprl/website/</a></em></p> + +<h3 id="q-what-is-redex-useful-for">Q. What is Redex useful for?</h3> + +<ol> + <li>declaring <a href="https://en.wikipedia.org/wiki/Regular_tree_grammar">regular tree grammars</a></li> + <li>defining <em>pattern</em>-based judgments and relations on <em>terms</em></li> + <li>testing properties of the above</li></ol> + +<p>More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.</p> + +<h3 id="q-what-is-redex-not-useful-for">Q. What is Redex <strong>not</strong> useful for?</h3> + +<p>Proving theorems about a grammar, judgment, or relation.</p> + +<h3 id="q-what-is-a-term">Q. What is a <em>term</em>?</h3> + +<p>Informally, a term is:</p> + +<ul> + <li>a Redex &ldquo;atom&rdquo;, or</li> + <li>an object that represents a sequence of characters.</li></ul> + +<p>More formally, a term is the result of evaluating <strong>(term X)</strong>, where <strong>X</strong> is any syntactically-correct Racket expression.</p> + +<p>Examples:</p> + +<pre><code>$ racket +Welcome to Racket v6.10.0.3. +&gt; (require redex/reduction-semantics) +&gt; (term 42) +42 +&gt; (term (+ 2 2)) +'(+ 2 2) +&gt; (term ("hello" world (#false))) +'("hello" world (#f))</code></pre> + +<p>Some terms may look strange. That&rsquo;s OK, because a term by itself has no meaning.</p> + +<p>Terms can refer to previously-defined values using the <strong>unquote</strong> escape.</p> + +<pre><code>&gt; (define x (term 42)) +&gt; (term (+ 2 x)) +'(+ 2 x) +&gt; (term (+ ,x (unquote x))) +'(+ 42 42)</code></pre> + +<h3 id="q-what-is-a-redex-model">Q. What is a <em>Redex model</em>?</h3> + +<p>A Redex model is collection of tools for working with terms. The tools may include:</p> + +<ul> + <li><em>languages</em>, to define a grammar for terms</li> + <li><em>judgments</em>, to describe properties of terms or relations between terms</li> + <li><em>metafunctions</em>, to transform terms</li> + <li><em>reduction relations</em>, to define a term rewriting system</li></ul> + +<p>The goal of these tools is to encode a &ldquo;real thing&rdquo; (maybe, a programming language) using Redex terms.</p> + +<h3 id="q-what-is-a-language">Q. What is a language?</h3> + +<p>A Redex <em>language</em> is a named set of non-terminals, <em>patterns</em>, and <em>binding forms</em>. For example, a Redex model of the natural numbers might start with this language:</p> + +<pre><code>(define-language nat + [N ::= Zero + (Plus1 N)])</code></pre> + +<ul> + <li>the name of the language is <strong>nat</strong></li> + <li>the non-terminal <strong>N</strong> is associated with two patterns: <strong>Zero</strong> and <strong>(Plus1 N)</strong></li> + <li>there are no <em>binding forms</em></li></ul> + +<p>Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.</p> + +<p>The non-terminal <strong>N</strong> describes all terms that are either:</p> + +<ol> + <li>the symbol <strong>Zero</strong></li> + <li>lists of the form <strong>(Plus1 N)</strong>, where <strong>N</strong> is either <strong>Zero</strong> or another &ldquo;Plus1&rdquo;</li></ol> + +<p>For example,</p> + +<pre><code>(term Zero) +(term (Plus1 Zero)) +(term (Plus1 (Plus1 Zero))) +(term (Plus1 (Plus1 (Plus1 Zero)))) +;; .... and so on</code></pre> + +<p>If a language has binding forms, then some terms can introduce names. See the question on <em>binding forms</em> (below) for an example.</p> + +<h3 id="q-what-is-a-pattern">Q. What is a pattern?</h3> + +<p>A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains <em>named non-terminals</em> from the language, then you can ask whether a Redex term matches the pattern.</p> + +<p>A <em>named non-terminal</em> for a language <strong>L</strong> is an identifier made of: (1) a non-terminal from <strong>L</strong>, (2) an underscore (<strong>_</strong>), and (3) any other identifier. See the FAQ entry below.</p> + +<p>For example, <strong>(redex-match? L p t)</strong> returns <strong>#true</strong> if the term <strong>t</strong> matches the pattern <strong>p</strong> relative to the language <strong>L</strong>.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(redex-match? nat N_some-name (term Zero)) +;; #true +(redex-match? nat (Plus1 N_a) (term Zero)) +;; #false +(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; #true</code></pre> + +<p>If <strong>(redex-match? L p t)</strong> is <strong>#true</strong>, then <strong>(redex-match L p t)</strong> shows how named non-terminals in the pattern bind to subterms of <strong>t</strong>.</p> + +<pre><code>(redex-match nat N_0 (term Zero)) +;; (list (match (list (bind 'N_0 'Zero)))) +(redex-match nat (Plus1 N_0) (term Zero)) +;; #f +(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))</code></pre> + +<h3 id="q-what-is-a-named-non-terminal">Q. What is a named non-terminal?</h3> + +<p>A named non-terminal in a language <strong>L</strong> is an identifier of the form <strong>X_Y</strong>, where:</p> + +<ul> + <li><strong>X</strong> is a non-terminal from <strong>L</strong></li> + <li><strong>Y</strong> is any identifier</li></ul> + +<p>The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.</p> + +<pre><code>(define-language trees + [binary-tree ::= Leaf + (Node binary-tree binary-tree)]) + +(redex-match trees + (Node binary-tree_left binary-tree_right) + (term (Node Leaf (Node Leaf Leaf)))) +;; (list +;; (match +;; (list (bind 'binary-tree_left 'Leaf) +;; (bind 'binary-tree_right '(Node Leaf Leaf)))))</code></pre> + +<h3 id="q-what-else-can-patterns-express">Q. What else can patterns express?</h3> + +<p>Redex patterns may contain special identifiers to guide pattern-matching. For instance:</p> + +<ul> + <li>The <strong>_</strong> pattern matches any term (and does not bind).</li> + <li>A pattern <strong>(p &hellip;)</strong> matches any sequence whose elements match the pattern <strong>p</strong>. If the pattern <strong>p</strong> is a named non-terminal, then the non-terminal binds a sequence of subterms.</li></ul> + +<p>Examples:</p> + +<pre><code>(redex-match? nat (Plus1 _) (term (Plus1 9))) +;; #true +(redex-match? nat (N_0 ...) (term ())) +;; #true +(redex-match? nat (N_0 ...) (term (Zero))) +;; #true +(redex-match nat (N_0 ...) (term (Zero Zero Zero))) +;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))</code></pre> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29">the Redex reference</a> for the full pattern language, including the <em>named and unique non-terminals</em> of the form <strong>X_!_Y</strong>.</p> + +<h3 id="q-what-can-patterns-not-express">Q. What can patterns <strong>not</strong> express?</h3> + +<ul> + <li>Disjunctions of patterns, e.g., &ldquo;number or boolean&rdquo;. (Use a language non-terminal.)</li> + <li>Negations of patterns. (Compose <strong>not</strong> with <strong>redex-match?</strong>.)</li> + <li>Some non-regular patterns. (Named dots <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29"><code>..._N</code></a> or <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-language%29%29"><code>define-language</code></a> with a side condition may be able to help.)</li></ul> + +<h3 id="q-what-is-a-judgment">Q. What is a judgment?</h3> + +<p>A Redex <em>judgment</em> form defines a relation on terms. The relation is defined by a set of inference rules.</p> + +<p>Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:</p> + +<ul> + <li>well-formedness conditions (i.e., whether a term contains free variables)</li> + <li>type checking rules</li> + <li>type inference rules</li> + <li>evaluation relations</li></ul> + +<p>Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(define-judgment-form nat + #:mode (N= I I) + #:contract (N= N N) + [ + --- Zero= + (N= Zero Zero)] + [ + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1) + (N= N_0-- N_1--) + --- Plus1= + (N= N_0 N_1)])</code></pre> + +<ol> + <li>the language is <strong>nat</strong>; Redex uses the language to interpret patterns</li> + <li>the mode is <strong>(N= I I)</strong>; this means <strong>N=</strong> is the name of a judgment that expects two input terms (or, <strong>N=</strong> is a binary relation on terms)</li> + <li>the contract is <strong>(N= N N)</strong>; in other words, <strong>N=</strong> expects two terms that match the <strong>N</strong> non-terminal from the <strong>nat</strong> language</li> + <li>there are two inference rules, named <strong>Zero=</strong> and <strong>Plus1=</strong></li> + <li>the <strong>Zero=</strong> rule states that <strong>(N= Zero Zero)</strong> always holds</li> + <li>the <strong>Plus1=</strong> rule states that <strong>(N= N_0 N_1)</strong> holds if <strong>N_0</strong> and <strong>N_1</strong> are both <strong>Plus1</strong> terms whose contents are related by <strong>N=</strong></li></ol> + +<p>The <strong>where</strong> clauses are <em>guards</em>. When Redex tries to apply a rule with premises of the form <strong>(where pattern term)</strong>, it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-judgment-form%29%29">the Redex reference</a> for more.</p> + +<pre><code>(judgment-holds (N= Zero Zero)) +;; #true +(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Note: the inference rules form a <em>set</em>, not a <em>sequence</em>. So when you ask Redex whether <strong>(judgment-holds (N= Zero Zero))</strong>, it applies all rules that match <strong>(N= Zero Zero)</strong>. For <strong>N=</strong> this is just one rule, but in general it could be many rules.</p> + +<h3 id="q-what-is-a-judgment-form-mode">Q. What is a judgment form <strong>#:mode</strong>?</h3> + +<p>A <strong>#:mode</strong> declaration expects a list of the form <strong>(id pos-use &hellip;)</strong>, where <strong>id</strong> is an identifier and each <strong>pos-use</strong> is either <strong>I</strong> or <strong>O</strong>. These declarations say four things:</p> + +<ol> + <li><strong>id</strong> is the name of a new judgment form</li> + <li><strong>id</strong> expects <strong>N</strong> arguments, where <strong>N</strong> is the number of <strong>pos-use</strong> symbols</li> + <li><strong>id</strong> expects an <em>input</em> at each position where the mode contains an <strong>I</strong></li> + <li><strong>id</strong> produces an <em>output</em> at each position where the mode contains an <strong>O</strong></li></ol> + +<p>For example, a type inference judgment may take an expression as input and output a type. Here&rsquo;s a fast and easy type inference judgment for arithmetic expressions. Given any term <strong>e_0</strong>, the judgment outputs the type <strong>Int</strong>.</p> + +<pre><code>(define-language Arith + (e ::= integer (e + e)) + (τ ::= Int)) + +(define-judgment-form Arith + #:mode (infer-type I O) + #:contract (infer-type e τ) + [ + --- T-Int + (infer-type e_0 Int)])</code></pre> + +<h3 id="q-what-can-judgments-not-express">Q. What can judgments <strong>not</strong> express?</h3> + +<p>Redex does not support inference rules that require guessing.</p> + +<p>One example of this is a transitivity rule: "<strong>τ_0</strong> is related to <strong>τ_2</strong> if there exists a <strong>τ_1</strong> such that <strong>τ_0</strong> is related to <strong>τ_1</strong> and <strong>τ_1</strong> is related to <strong>τ_2</strong>". The following example tries to define a transitive subtyping (<strong>&lt;:</strong>) relation, but Redex rejects the <strong>S-Trans</strong> rule.</p> + +<pre><code>(define-language SomeTypes + (τ ::= (→ τ τ) Integer)) + +(define-judgment-form SomeTypes + #:mode (&lt;: I I) + #:contract (&lt;: τ τ) + [ + (&lt;: τ_0 τ_1) + (&lt;: τ_1 τ_2) + --- S-Trans + (&lt;: τ_0 τ_2)] + [ + --- S-Refl + (&lt;: τ_0 τ_0)] + [ + (&lt;: τ_dom-1 τ_dom-0) + (&lt;: τ_cod-0 τ_cod-1) + --- S-Arrow + (&lt;: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])</code></pre> + +<p>The error is that in the rule <strong>S-Trans</strong>, the named non-terminal <strong>τ_1</strong> appears in an input position but is not bound to a term.</p> + +<h3 id="q-what-is-a-metafunction">Q. What is a metafunction?</h3> + +<p>A metafunction is a term-level function on terms.</p> + +<p>Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.</p> + +<p>Here is a metafunction that returns <strong>#true</strong> when given two equal natural numbers. The definition is similar to the <strong>N=</strong> judgment form.</p> + +<pre><code>(define-metafunction nat + N=? : N N -&gt; boolean + [(N=? Zero Zero) + #true] + [(N=? N_0 N_1) + (N=? N_0-- N_1--) + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1)] + [(N=? N_0 N_1) + #false])</code></pre> + +<ul> + <li>the metafunction is named <strong>N=?</strong></li> + <li>its contract is <strong>N N -&gt; boolean</strong>, this means <strong>N=?</strong> expects 2 terms that match the <strong>N</strong> pattern and returns a term that matches the pattern <strong>boolean</strong></li> + <li>there are three cases; the second case is guarded by two <strong>where</strong> clauses</li></ul> + +<p>Any occurrence of <strong>(N=? &hellip;.)</strong> in any term is evaluated.</p> + +<pre><code>(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(term ((N=? Zero Zero) Zero)) +;; '(#true Zero) +(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Any occurrence of <strong>N=?</strong> outside a <strong>term</strong> is an error.</p> + +<p>Metafunction <strong>where</strong>-clauses are analogous to judgment form <strong>where</strong>-clauses. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._metafunction%29">the Redex reference</a> for more.</p> + +<p>Note: the cases in a metafunction form a <em>sequence</em>, not a <em>set</em>. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.</p> + +<h3 id="q-should-i-use-a-metafunction-or-a-judgment-form">Q. Should I use a metafunction or a judgment form?</h3> + +<p>Use a judgment form.</p> + +<p>Metafunctions are handy, but judgments are easier to read and debug and maintain.</p> + +<h3 id="q-what-is-a-reduction-relation">Q. What is a reduction relation?</h3> + +<p>A reduction relation is a set of term-rewriting rules.</p> + +<p>Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.</p> + +<ul> + <li>The language tells Redex how to interpret patterns.</li> + <li>The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.</li> + <li>The rewrite rules have the form <strong>(&mdash;&gt; term term guard &hellip;)</strong>. The term on the left is the input, the term on the right is the output.</li></ul> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._reduction-relation%29%29">the Redex reference</a> for a full description of the guards.</p> + +<p>The preferred way to define a reduction relation is to define a language that includes three non-terminals:</p> + +<ol> + <li>a non-terminal for the domain of the reduction relation</li> + <li>a non-terminal for a <em>subset</em> of the domain that cannot reduce further</li> + <li>a non-terminal for evaluation contexts</li></ol> + +<p>An evaluation context is a term that contains a <strong>hole</strong>. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite &mdash; in particular, the sub-term that matches the <strong>hole</strong>.</p> + +<p>In the following example, <strong>bexp</strong> is the domain of a reduction relation. A <strong>bexp</strong> term represents a boolean expression, which can be <strong>#true</strong> or <strong>#false</strong> or a conjunction of expressions or a disjunction of expressions. The boolean expressions <strong>#true</strong> and <strong>#false</strong> are also values (<strong>val</strong>); these cannot reduce further. The non-terminal <strong>E</strong> is for evaluation contexts.</p> + +<pre><code>(define-language Bool + (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp)) + (val ::= #true #false) + (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E))) + +(define step + (reduction-relation Bool + #:domain bexp + [--&gt; (in-hole E (val_lhs ∧ val_rhs)) + (in-hole E val_new) + ∧-step + (where val_new ,(and (term val_lhs) (term val_rhs)))] + [--&gt; (in-hole E (val_lhs ∨ val_rhs)) + (in-hole E val_new) + ∨-step + (where val_new ,(or (term val_lhs) (term val_rhs)))]))</code></pre> + +<p>The function <strong>apply-reduction-relation</strong> applies a reduction relation to a term and returns a list of ways that the term can step.</p> + +<pre><code>(apply-reduction-relation step (term #true)) +;; '() +(apply-reduction-relation step (term (#true ∧ #true))) +;; '(#true) +(apply-reduction-relation step (term (#true ∧ #false))) +;; '(#false) +(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true))) +;; '((#true ∧ #true))</code></pre> + +<p>Three things about the reduction relation <strong>step</strong>:</p> + +<ol> + <li>Using <strong>in-hole</strong> on the first argument of <strong>&mdash;&gt;</strong> searches a term for a subterm that Redex can apply a reduction rule to.</li> + <li>Using <strong>in-hole</strong> on the second argument of <strong>&mdash;&gt;</strong> puts a new value back into the <strong>hole</strong> in the evaluation context.</li> + <li>The unquote operator (<strong>,</strong>) escapes to &ldquo;Racket mode&rdquo; (see below) to evaluate a conjunction or disjunction.</li></ol> + +<p>A judgment or metafunction is a formal alternative to &ldquo;escaping to Racket&rdquo;, but escaping can be convenient.</p> + +<p>Note: the cases in a reduction relation form a <em>set</em>, not a <em>sequence</em>. If more than one case matches, Redex applies them all.</p> + +<h3 id="q-what-is-racket-mode-what-is-redex-mode">Q. What is &ldquo;Racket mode&rdquo;? What is &ldquo;Redex mode&rdquo;?</h3> + +<p>Code in a Redex model is sometimes evaluated in &ldquo;Racket mode&rdquo; and sometimes evaluated in &ldquo;Redex mode&rdquo;. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.</p> + +<p>Key points:</p> + +<ol> + <li>A Redex program starts in Racket mode.</li> + <li>The <strong>X</strong> in <strong>(term X)</strong> is evaluated in Redex mode &hellip;</li> + <li>&hellip; unless <strong>X</strong> contains unquoted sub-expressions. Unquoting escapes to Racket mode &hellip;</li> + <li>&hellip; and <strong>term</strong>s inside an unquoted sub-expression are evaluated in Redex mode.</li></ol> + +<p>In other words, <strong>term</strong> enters Redex mode and <strong>unquote</strong> (<strong>,</strong>) escapes back to Racket.</p> + +<p>Redex implicitly switches to Redex mode in a few other places, for example:</p> + +<ul> + <li>the right-side of a <strong>where</strong> clause is in Redex mode</li> + <li>the result of a metafunction is in Redex mode</li></ul> + +<p>When in doubt, try using an <strong>unquote</strong>. Redex will raise an exception if it finds an unquote in Racket mode.</p> + +<h3 id="q-are-side-conditions-evaluated-in-racket-mode-or-redex-mode">Q. Are <strong>side-condition</strong>s evaluated in &ldquo;Racket mode&rdquo; or &ldquo;Redex mode&rdquo;?</h3> + +<p>A <strong>(side-condition e)</strong> sometimes evaluates <strong>e</strong> as a Racket expression and sometimes evaluates <strong>e</strong> as a Redex expression.</p> + +<ul> + <li>reduction relations and metafunctions expect a <strong>Racket</strong> expression</li> + <li>judgments expect a <strong>Redex</strong> expression</li></ul> + +<h3 id="q-what-is-a-binding-form">Q. What is a binding form?</h3> + +<p>In the lambda calculus, <strong>λ</strong>-terms bind variables. A term <strong>(λ x M)</strong> means that any free occurrence of <strong>x</strong> in the sub-term <strong>M</strong> refers to the <strong>x</strong> from the <strong>λ</strong>-term.</p> + +<p>Redex can express this idea with a binding form.</p> + +<pre><code>(define-language Λ + [e ::= (e e) x (λ x e)] + [x ::= variable-not-otherwise-mentioned] + #:binding-forms + (λ x_0 e_0 #:refers-to x_0))</code></pre> + +<p>Note: all the non-terminals in a language must be defined before the <strong>#:binding-forms</strong> keyword. If a non-terminal definition appears after the <strong>#:binding-forms</strong> keyword, then Redex will interpret the &ldquo;definition&rdquo; as a binding form.</p> + +<p>Binding forms work together with Redex&rsquo;s functions for substitution and alphabetic equivalence.</p> + +<pre><code>(alpha-equivalent? Λ + (term (λ x x)) + (term (λ y y)))) +;; #true + +(define-metafunction Λ + test-substitute : e -&gt; e + [(test-substitute (λ x_0 e_0)) + (substitute e_0 x_0 y)]) +(term (test-substitute (λ z (z z)))) +;; '(y y)</code></pre> + +<h3 id="q-what-is--what-is-">Q. What is <strong>&hellip;</strong>? What is <strong>&hellip;.</strong>?</h3> + +<p>Three dots (<strong>&hellip;</strong>) is for building patterns. If <strong>p</strong> is a pattern then <strong>(p &hellip;)</strong> matches any list whose elements all match <strong>p</strong>.</p> + +<pre><code>(define-language L) +(redex-match? L (number ... boolean ...) (term (1 2 #true #true))) +;; #true</code></pre> + +<p>Four dots (<strong>&hellip;.</strong>) may be used in <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-extended-language%29%29"><strong>define-extended-language</strong></a> to extend a previously-defined non-terminal.</p> + +<pre><code>(define-language C + (keyword ::= auto break case)) +(define-extended-language C++ + C + (keyword ::= .... class)) + +(redex-match? C keyword (term auto)) +;; #true +(redex-match? C keyword (term class)) +;; #false +(redex-match? C++ keyword (term auto)) +;; #true +(redex-match? C++ keyword (term class)) +;; #true</code></pre> + +<h3 id="q-where-to-learn-more-about-redex">Q. Where to learn more about Redex?</h3> + +<p>&ldquo;Critical path&rdquo; resources:</p> + +<ul> + <li>Code examples for this post: <a href="https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt">https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt</a></li> + <li>Redex documentation: <a href="http://docs.racket-lang.org/redex/index.html">http://docs.racket-lang.org/redex/index.html</a></li> + <li><a href="https://dvanhorn.github.io/redex-aam-tutorial/"><em>An Introduction to Redex with Abstracting Abstract Machines</em></a> by David Van Horn</li> + <li><a href="https://www.leafac.com/playing-the-game-with-plt-redex/"><em>Playing the Game with PLT Redex</em></a> by Leandro Facchinetti</li> + <li><a href="https://williamjbowman.com/doc/experimenting-with-redex/index.html"><em>Experimenting with Languages in Redex</em></a> by William J. Bowman</li></ul> + +<p>&ldquo;Procrastination&rdquo; resources:</p> + +<ul> + <li><a href="http://tata.gforge.inria.fr/"><em>Tree Automata Techniques and Applications</em></a></li> + <li><a href="http://lamport.azurewebsites.net/pubs/lamport-types.pdf"><em>Should your Specification Language be Typed?</em></a></li> + <li>Redex source code (see <code>redex-lib/</code>): <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + + PLT Redex: mf-apply + + urn:http-prl-ccs-neu-edu:-blog-2017-03-03-plt-redex-mf-apply + 2017-03-03T08:54:20Z + 2017-03-03T08:54:20Z + + Ben Greenman + +<p>The <code>mf-apply</code> keyword is for checked metafunction application in PLT Redex. In other words, <code>(mf-apply f x)</code> is just like <code>(f x)</code>, but errors if <code>f</code> is not a previously-defined metafunction.</p> + +<p>Also, consider applying to attend <em>The Racket School of Semantics and Languages</em> in Salt Lake City this summer: <a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></p> +<!-- more--> + +<h2 id="metafunctions-vs-list-patterns">Metafunctions vs. List Patterns</h2> + +<p>Have you used PLT Redex? Good! Maybe this has happened to you:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span> +<span class="normal">25</span> +<span class="normal">26</span> +<span class="normal">27</span> +<span class="normal">28</span> +<span class="normal">29</span> +<span class="normal">30</span> +<span class="normal">31</span> +<span class="normal">32</span> +<span class="normal">33</span> +<span class="normal">34</span> +<span class="normal">35</span> +<span class="normal">36</span> +<span class="normal">37</span> +<span class="normal">38</span> +<span class="normal">39</span> +<span class="normal">40</span> +<span class="normal">41</span> +<span class="normal">42</span> +<span class="normal">43</span> +<span class="normal">44</span> +<span class="normal">45</span> +<span class="normal">46</span> +<span class="normal">47</span> +<span class="normal">48</span> +<span class="normal">49</span> +<span class="normal">50</span> +<span class="normal">51</span> +<span class="normal">52</span> +<span class="normal">53</span> +<span class="normal">54</span> +<span class="normal">55</span> +<span class="normal">56</span> +<span class="normal">57</span> +<span class="normal">58</span> +<span class="normal">59</span> +<span class="normal">60</span> +<span class="normal">61</span> +<span class="normal">62</span> +<span class="normal">63</span> +<span class="normal">64</span> +<span class="normal">65</span> +<span class="normal">66</span> +<span class="normal">67</span> +<span class="normal">68</span> +<span class="normal">69</span> +<span class="normal">70</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 1. You define a language</span> +<span class="p">(</span><span class="n">define-language</span> <span class="n">STLC</span> + <span class="p">[</span><span class="n">V</span> <span class="n">::=</span> <span class="n">integer</span> <span class="n">boolean</span> <span class="n">C</span><span class="p">]</span> + <span class="p">[</span><span class="n">C</span> <span class="n">::=</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">Λ</span> <span class="n">::=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)]</span> + <span class="p">[</span><span class="n">M</span> <span class="n">::=</span> <span class="p">(</span><span class="n">M</span> <span class="n">M</span><span class="p">)</span> <span class="n">V</span> <span class="n">Λ</span> <span class="n">x</span><span class="p">]</span> + <span class="p">[</span><span class="n">τ</span> <span class="n">::=</span> <span class="n">Int</span> <span class="n">Bool</span> <span class="p">(</span><span class="n">τ</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">ρ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">V</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">Γ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">τ</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">x</span> <span class="n">::=</span> <span class="n">variable-not-otherwise-mentioned</span><span class="p">]</span> + <span class="kd">#:binding-forms</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span> <span class="kd">#:refers-to</span> <span class="n">x</span><span class="p">))</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 2. You define a few metafunctions</span> +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;lam</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">Λ</span> + <span class="p">[(</span><span class="n">closure-&gt;lam</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">Λ</span><span class="p">])</span> + +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;env</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">ρ</span> + <span class="p">[(</span><span class="n">closure-&gt;env</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">ρ</span><span class="p">])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 3. You try defining a judgment form . . .</span> +<span class="p">(</span><span class="n">define-judgment-form</span> <span class="n">STLC</span> + <span class="kd">#:mode</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">I</span> <span class="n">O</span><span class="p">)</span> + <span class="kd">#:contract</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Var</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">x</span> <span class="p">(</span><span class="n">x</span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_0</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_1</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">---</span> <span class="n">FVS-App</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">M_0</span> <span class="n">M_1</span><span class="p">)</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x_0</span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)</span> <span class="n">Λ</span><span class="p">)</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-remove))" style="color: inherit">set-remove</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="n">x_0</span><span class="p">)))</span> + <span class="n">---</span> <span class="n">FVS-Λ</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Integer</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">integer_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Boolean</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">boolean_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">closure-&gt;lam</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-subtract))" style="color: inherit">set-subtract</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))))</span> + <span class="n">---</span> <span class="n">FVS-Closure</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">C</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 4. You test the judgment, and it mysteriously fails</span> +<span class="p">(</span><span class="n">judgment-holds</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">closure</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">())</span> + <span class="p">()))</span> +<span class="c1">;; ==&gt; #f</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><strong>WHAT HAPPENED??!</strong></p> + +<p>The problem is this line in the <code>FVS-Closure</code> rule:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which checks that the list <code>(closure-env C)</code> (whose first element is the symbol <code>closure-env</code> and second element is the symbol <code>C</code>) matches the pattern <code>((x_1 τ_1) ...)</code>.</p> + +<p>Right.</p> + +<p>Of course you meant to apply the metafunction <code>closure-&gt;env</code> but made a typo. And since the syntax for metafunction application is the same as the syntax for building a list, Redex doesn&rsquo;t report an error.</p> + +<p>We can fix this code with the new <a href="https://www.cs.utah.edu/plt/snapshots/current/doc/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._mf-apply%29%29"><code>mf-apply</code></a> keyword (available on <a href="https://github.com/racket/racket">GitHub</a> or in a <a href="https://www.cs.utah.edu/plt/snapshots/">snapshot build</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running <code>raco make</code> now gives a compile-time error.</p> + +<pre><code> term: expected a previously defined metafunction + at: closure-env + in: (mf-apply closure-env C)</code></pre> + +<h3 id="but-i-still-need-to-type-mf-apply-correctly">But I still need to type <code>mf-apply</code> correctly!</h3> + +<p>Leif Andersen says:</p> + +<blockquote> + <p>I should point out that this has the issue of you still need to type <code>mf-apply</code> correctly. ;)</p></blockquote> + +<p>That is, if you accidentally write:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-applu</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then the code compiles, thinking you intend to match a list of three elements against the pattern.</p> + +<p>Never fear, there are at least two solutions.</p> + +<h4 id="solution-1-rename-mf-apply">Solution 1: rename <code>mf-apply</code></h4> + +<p>A simple fix is to rename the <code>mf-apply</code> keyword to something shorter (and harder to mis-type):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._rename-in))" style="color: inherit">rename-in</a></span> <span class="n">redex</span> + <span class="p">[</span><span class="n">mf-apply</span> <span class="n">MF</span><span class="p">]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="solution-2-the-mf-apply-lang-extension">Solution 2: the <code>mf-apply</code> lang extension</h4> + +<p>A fancier solution is to install the <code>mf-apply</code> meta-language.</p> + +<pre><code> $ raco pkg install mf-apply</code></pre> + +<p>This language updates the <a href="http://docs.racket-lang.org/reference/readtables.html#%28tech._readtable%29"><em>readtable</em></a> to interpret S-expressions that begin with <code>#{</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="o">#</span><span class="p">{</span><span class="ss">f</span> <span class="ss">x</span> <span class="ss"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>as a metafunction application:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">f</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>You the programmer only needs to write the <code>#{....}</code> syntax.</p> + +<p>Source code is on GitHub:</p> + +<ul> + <li><a href="https://github.com/bennn/mf-apply">https://github.com/bennn/mf-apply</a></li></ul> + +<p>(It&rsquo;s the simplest lang-extension I know of)</p> + +<h2 id="what-is-plt-redex">What is PLT Redex?</h2> + +<p>PLT Redex is a library for semantics engineering. One of my favorite Redex features is it implements capture-avoiding substitution and α-equivalence for any language with a <code>#:binding-forms</code> specification (such as STLC, above).</p> + +<p>You can read more:</p> + +<ul> + <li>in the &ldquo;Amb&rdquo; tutorial: <a href="http://docs.racket-lang.org/redex/tutorial.html">http://docs.racket-lang.org/redex/tutorial.html</a></li> + <li>in the &ldquo;Long Tutorial&rdquo;: <a href="http://docs.racket-lang.org/redex/redex2015.html">http://docs.racket-lang.org/redex/redex2015.html</a></li> + <li>in the Redex reference manual: <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html">http://docs.racket-lang.org/redex/The_Redex_Reference.html</a></li> + <li>on the PLT Redex website: <a href="https://redex.racket-lang.org/">https://redex.racket-lang.org/</a></li> + <li>on GitHub: <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + +<p>And if you act now, you can become a <em>Redexan</em> between July 10 and July 14 at the summer school in Salt Lake City, Utah:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/PLT-Redex.rss.xml b/blog/feeds/PLT-Redex.rss.xml new file mode 100644 index 00000000..58fedba8 --- /dev/null +++ b/blog/feeds/PLT-Redex.rss.xml @@ -0,0 +1,849 @@ + + + + PRL Blog: Posts tagged 'PLT Redex' + PRL Blog: Posts tagged 'PLT Redex' + http://prl.ccs.neu.edu/blog/tags/PLT-Redex.html + Mon, 25 Sep 2017 23:39:16 UT + Mon, 25 Sep 2017 23:39:16 UT + 1800 + + PLT Redex FAQ + http://prl.ccs.neu.edu/blog/2017/09/25/plt-redex-faq/?utm_source=PLT-Redex&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-25-plt-redex-faq + Mon, 25 Sep 2017 23:39:16 UT + Ben Greenman, Sam Caldwell + +<p>A short guide to Redex concepts, conventions, and common mistakes.</p> +<!-- more--> + +<hr /> + +<p><em>To contribute to this FAQ, submit issues and pull requests to: <a href="https://github.com/nuprl/website/">https://github.com/nuprl/website/</a></em></p> + +<h3 id="q-what-is-redex-useful-for">Q. What is Redex useful for?</h3> + +<ol> + <li>declaring <a href="https://en.wikipedia.org/wiki/Regular_tree_grammar">regular tree grammars</a></li> + <li>defining <em>pattern</em>-based judgments and relations on <em>terms</em></li> + <li>testing properties of the above</li></ol> + +<p>More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.</p> + +<h3 id="q-what-is-redex-not-useful-for">Q. What is Redex <strong>not</strong> useful for?</h3> + +<p>Proving theorems about a grammar, judgment, or relation.</p> + +<h3 id="q-what-is-a-term">Q. What is a <em>term</em>?</h3> + +<p>Informally, a term is:</p> + +<ul> + <li>a Redex &ldquo;atom&rdquo;, or</li> + <li>an object that represents a sequence of characters.</li></ul> + +<p>More formally, a term is the result of evaluating <strong>(term X)</strong>, where <strong>X</strong> is any syntactically-correct Racket expression.</p> + +<p>Examples:</p> + +<pre><code>$ racket +Welcome to Racket v6.10.0.3. +&gt; (require redex/reduction-semantics) +&gt; (term 42) +42 +&gt; (term (+ 2 2)) +'(+ 2 2) +&gt; (term ("hello" world (#false))) +'("hello" world (#f))</code></pre> + +<p>Some terms may look strange. That&rsquo;s OK, because a term by itself has no meaning.</p> + +<p>Terms can refer to previously-defined values using the <strong>unquote</strong> escape.</p> + +<pre><code>&gt; (define x (term 42)) +&gt; (term (+ 2 x)) +'(+ 2 x) +&gt; (term (+ ,x (unquote x))) +'(+ 42 42)</code></pre> + +<h3 id="q-what-is-a-redex-model">Q. What is a <em>Redex model</em>?</h3> + +<p>A Redex model is collection of tools for working with terms. The tools may include:</p> + +<ul> + <li><em>languages</em>, to define a grammar for terms</li> + <li><em>judgments</em>, to describe properties of terms or relations between terms</li> + <li><em>metafunctions</em>, to transform terms</li> + <li><em>reduction relations</em>, to define a term rewriting system</li></ul> + +<p>The goal of these tools is to encode a &ldquo;real thing&rdquo; (maybe, a programming language) using Redex terms.</p> + +<h3 id="q-what-is-a-language">Q. What is a language?</h3> + +<p>A Redex <em>language</em> is a named set of non-terminals, <em>patterns</em>, and <em>binding forms</em>. For example, a Redex model of the natural numbers might start with this language:</p> + +<pre><code>(define-language nat + [N ::= Zero + (Plus1 N)])</code></pre> + +<ul> + <li>the name of the language is <strong>nat</strong></li> + <li>the non-terminal <strong>N</strong> is associated with two patterns: <strong>Zero</strong> and <strong>(Plus1 N)</strong></li> + <li>there are no <em>binding forms</em></li></ul> + +<p>Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.</p> + +<p>The non-terminal <strong>N</strong> describes all terms that are either:</p> + +<ol> + <li>the symbol <strong>Zero</strong></li> + <li>lists of the form <strong>(Plus1 N)</strong>, where <strong>N</strong> is either <strong>Zero</strong> or another &ldquo;Plus1&rdquo;</li></ol> + +<p>For example,</p> + +<pre><code>(term Zero) +(term (Plus1 Zero)) +(term (Plus1 (Plus1 Zero))) +(term (Plus1 (Plus1 (Plus1 Zero)))) +;; .... and so on</code></pre> + +<p>If a language has binding forms, then some terms can introduce names. See the question on <em>binding forms</em> (below) for an example.</p> + +<h3 id="q-what-is-a-pattern">Q. What is a pattern?</h3> + +<p>A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains <em>named non-terminals</em> from the language, then you can ask whether a Redex term matches the pattern.</p> + +<p>A <em>named non-terminal</em> for a language <strong>L</strong> is an identifier made of: (1) a non-terminal from <strong>L</strong>, (2) an underscore (<strong>_</strong>), and (3) any other identifier. See the FAQ entry below.</p> + +<p>For example, <strong>(redex-match? L p t)</strong> returns <strong>#true</strong> if the term <strong>t</strong> matches the pattern <strong>p</strong> relative to the language <strong>L</strong>.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(redex-match? nat N_some-name (term Zero)) +;; #true +(redex-match? nat (Plus1 N_a) (term Zero)) +;; #false +(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; #true</code></pre> + +<p>If <strong>(redex-match? L p t)</strong> is <strong>#true</strong>, then <strong>(redex-match L p t)</strong> shows how named non-terminals in the pattern bind to subterms of <strong>t</strong>.</p> + +<pre><code>(redex-match nat N_0 (term Zero)) +;; (list (match (list (bind 'N_0 'Zero)))) +(redex-match nat (Plus1 N_0) (term Zero)) +;; #f +(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))</code></pre> + +<h3 id="q-what-is-a-named-non-terminal">Q. What is a named non-terminal?</h3> + +<p>A named non-terminal in a language <strong>L</strong> is an identifier of the form <strong>X_Y</strong>, where:</p> + +<ul> + <li><strong>X</strong> is a non-terminal from <strong>L</strong></li> + <li><strong>Y</strong> is any identifier</li></ul> + +<p>The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.</p> + +<pre><code>(define-language trees + [binary-tree ::= Leaf + (Node binary-tree binary-tree)]) + +(redex-match trees + (Node binary-tree_left binary-tree_right) + (term (Node Leaf (Node Leaf Leaf)))) +;; (list +;; (match +;; (list (bind 'binary-tree_left 'Leaf) +;; (bind 'binary-tree_right '(Node Leaf Leaf)))))</code></pre> + +<h3 id="q-what-else-can-patterns-express">Q. What else can patterns express?</h3> + +<p>Redex patterns may contain special identifiers to guide pattern-matching. For instance:</p> + +<ul> + <li>The <strong>_</strong> pattern matches any term (and does not bind).</li> + <li>A pattern <strong>(p &hellip;)</strong> matches any sequence whose elements match the pattern <strong>p</strong>. If the pattern <strong>p</strong> is a named non-terminal, then the non-terminal binds a sequence of subterms.</li></ul> + +<p>Examples:</p> + +<pre><code>(redex-match? nat (Plus1 _) (term (Plus1 9))) +;; #true +(redex-match? nat (N_0 ...) (term ())) +;; #true +(redex-match? nat (N_0 ...) (term (Zero))) +;; #true +(redex-match nat (N_0 ...) (term (Zero Zero Zero))) +;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))</code></pre> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29">the Redex reference</a> for the full pattern language, including the <em>named and unique non-terminals</em> of the form <strong>X_!_Y</strong>.</p> + +<h3 id="q-what-can-patterns-not-express">Q. What can patterns <strong>not</strong> express?</h3> + +<ul> + <li>Disjunctions of patterns, e.g., &ldquo;number or boolean&rdquo;. (Use a language non-terminal.)</li> + <li>Negations of patterns. (Compose <strong>not</strong> with <strong>redex-match?</strong>.)</li> + <li>Some non-regular patterns. (Named dots <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29"><code>..._N</code></a> or <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-language%29%29"><code>define-language</code></a> with a side condition may be able to help.)</li></ul> + +<h3 id="q-what-is-a-judgment">Q. What is a judgment?</h3> + +<p>A Redex <em>judgment</em> form defines a relation on terms. The relation is defined by a set of inference rules.</p> + +<p>Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:</p> + +<ul> + <li>well-formedness conditions (i.e., whether a term contains free variables)</li> + <li>type checking rules</li> + <li>type inference rules</li> + <li>evaluation relations</li></ul> + +<p>Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(define-judgment-form nat + #:mode (N= I I) + #:contract (N= N N) + [ + --- Zero= + (N= Zero Zero)] + [ + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1) + (N= N_0-- N_1--) + --- Plus1= + (N= N_0 N_1)])</code></pre> + +<ol> + <li>the language is <strong>nat</strong>; Redex uses the language to interpret patterns</li> + <li>the mode is <strong>(N= I I)</strong>; this means <strong>N=</strong> is the name of a judgment that expects two input terms (or, <strong>N=</strong> is a binary relation on terms)</li> + <li>the contract is <strong>(N= N N)</strong>; in other words, <strong>N=</strong> expects two terms that match the <strong>N</strong> non-terminal from the <strong>nat</strong> language</li> + <li>there are two inference rules, named <strong>Zero=</strong> and <strong>Plus1=</strong></li> + <li>the <strong>Zero=</strong> rule states that <strong>(N= Zero Zero)</strong> always holds</li> + <li>the <strong>Plus1=</strong> rule states that <strong>(N= N_0 N_1)</strong> holds if <strong>N_0</strong> and <strong>N_1</strong> are both <strong>Plus1</strong> terms whose contents are related by <strong>N=</strong></li></ol> + +<p>The <strong>where</strong> clauses are <em>guards</em>. When Redex tries to apply a rule with premises of the form <strong>(where pattern term)</strong>, it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-judgment-form%29%29">the Redex reference</a> for more.</p> + +<pre><code>(judgment-holds (N= Zero Zero)) +;; #true +(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Note: the inference rules form a <em>set</em>, not a <em>sequence</em>. So when you ask Redex whether <strong>(judgment-holds (N= Zero Zero))</strong>, it applies all rules that match <strong>(N= Zero Zero)</strong>. For <strong>N=</strong> this is just one rule, but in general it could be many rules.</p> + +<h3 id="q-what-is-a-judgment-form-mode">Q. What is a judgment form <strong>#:mode</strong>?</h3> + +<p>A <strong>#:mode</strong> declaration expects a list of the form <strong>(id pos-use &hellip;)</strong>, where <strong>id</strong> is an identifier and each <strong>pos-use</strong> is either <strong>I</strong> or <strong>O</strong>. These declarations say four things:</p> + +<ol> + <li><strong>id</strong> is the name of a new judgment form</li> + <li><strong>id</strong> expects <strong>N</strong> arguments, where <strong>N</strong> is the number of <strong>pos-use</strong> symbols</li> + <li><strong>id</strong> expects an <em>input</em> at each position where the mode contains an <strong>I</strong></li> + <li><strong>id</strong> produces an <em>output</em> at each position where the mode contains an <strong>O</strong></li></ol> + +<p>For example, a type inference judgment may take an expression as input and output a type. Here&rsquo;s a fast and easy type inference judgment for arithmetic expressions. Given any term <strong>e_0</strong>, the judgment outputs the type <strong>Int</strong>.</p> + +<pre><code>(define-language Arith + (e ::= integer (e + e)) + (τ ::= Int)) + +(define-judgment-form Arith + #:mode (infer-type I O) + #:contract (infer-type e τ) + [ + --- T-Int + (infer-type e_0 Int)])</code></pre> + +<h3 id="q-what-can-judgments-not-express">Q. What can judgments <strong>not</strong> express?</h3> + +<p>Redex does not support inference rules that require guessing.</p> + +<p>One example of this is a transitivity rule: "<strong>τ_0</strong> is related to <strong>τ_2</strong> if there exists a <strong>τ_1</strong> such that <strong>τ_0</strong> is related to <strong>τ_1</strong> and <strong>τ_1</strong> is related to <strong>τ_2</strong>". The following example tries to define a transitive subtyping (<strong>&lt;:</strong>) relation, but Redex rejects the <strong>S-Trans</strong> rule.</p> + +<pre><code>(define-language SomeTypes + (τ ::= (→ τ τ) Integer)) + +(define-judgment-form SomeTypes + #:mode (&lt;: I I) + #:contract (&lt;: τ τ) + [ + (&lt;: τ_0 τ_1) + (&lt;: τ_1 τ_2) + --- S-Trans + (&lt;: τ_0 τ_2)] + [ + --- S-Refl + (&lt;: τ_0 τ_0)] + [ + (&lt;: τ_dom-1 τ_dom-0) + (&lt;: τ_cod-0 τ_cod-1) + --- S-Arrow + (&lt;: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])</code></pre> + +<p>The error is that in the rule <strong>S-Trans</strong>, the named non-terminal <strong>τ_1</strong> appears in an input position but is not bound to a term.</p> + +<h3 id="q-what-is-a-metafunction">Q. What is a metafunction?</h3> + +<p>A metafunction is a term-level function on terms.</p> + +<p>Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.</p> + +<p>Here is a metafunction that returns <strong>#true</strong> when given two equal natural numbers. The definition is similar to the <strong>N=</strong> judgment form.</p> + +<pre><code>(define-metafunction nat + N=? : N N -&gt; boolean + [(N=? Zero Zero) + #true] + [(N=? N_0 N_1) + (N=? N_0-- N_1--) + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1)] + [(N=? N_0 N_1) + #false])</code></pre> + +<ul> + <li>the metafunction is named <strong>N=?</strong></li> + <li>its contract is <strong>N N -&gt; boolean</strong>, this means <strong>N=?</strong> expects 2 terms that match the <strong>N</strong> pattern and returns a term that matches the pattern <strong>boolean</strong></li> + <li>there are three cases; the second case is guarded by two <strong>where</strong> clauses</li></ul> + +<p>Any occurrence of <strong>(N=? &hellip;.)</strong> in any term is evaluated.</p> + +<pre><code>(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(term ((N=? Zero Zero) Zero)) +;; '(#true Zero) +(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Any occurrence of <strong>N=?</strong> outside a <strong>term</strong> is an error.</p> + +<p>Metafunction <strong>where</strong>-clauses are analogous to judgment form <strong>where</strong>-clauses. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._metafunction%29">the Redex reference</a> for more.</p> + +<p>Note: the cases in a metafunction form a <em>sequence</em>, not a <em>set</em>. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.</p> + +<h3 id="q-should-i-use-a-metafunction-or-a-judgment-form">Q. Should I use a metafunction or a judgment form?</h3> + +<p>Use a judgment form.</p> + +<p>Metafunctions are handy, but judgments are easier to read and debug and maintain.</p> + +<h3 id="q-what-is-a-reduction-relation">Q. What is a reduction relation?</h3> + +<p>A reduction relation is a set of term-rewriting rules.</p> + +<p>Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.</p> + +<ul> + <li>The language tells Redex how to interpret patterns.</li> + <li>The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.</li> + <li>The rewrite rules have the form <strong>(&mdash;&gt; term term guard &hellip;)</strong>. The term on the left is the input, the term on the right is the output.</li></ul> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._reduction-relation%29%29">the Redex reference</a> for a full description of the guards.</p> + +<p>The preferred way to define a reduction relation is to define a language that includes three non-terminals:</p> + +<ol> + <li>a non-terminal for the domain of the reduction relation</li> + <li>a non-terminal for a <em>subset</em> of the domain that cannot reduce further</li> + <li>a non-terminal for evaluation contexts</li></ol> + +<p>An evaluation context is a term that contains a <strong>hole</strong>. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite &mdash; in particular, the sub-term that matches the <strong>hole</strong>.</p> + +<p>In the following example, <strong>bexp</strong> is the domain of a reduction relation. A <strong>bexp</strong> term represents a boolean expression, which can be <strong>#true</strong> or <strong>#false</strong> or a conjunction of expressions or a disjunction of expressions. The boolean expressions <strong>#true</strong> and <strong>#false</strong> are also values (<strong>val</strong>); these cannot reduce further. The non-terminal <strong>E</strong> is for evaluation contexts.</p> + +<pre><code>(define-language Bool + (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp)) + (val ::= #true #false) + (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E))) + +(define step + (reduction-relation Bool + #:domain bexp + [--&gt; (in-hole E (val_lhs ∧ val_rhs)) + (in-hole E val_new) + ∧-step + (where val_new ,(and (term val_lhs) (term val_rhs)))] + [--&gt; (in-hole E (val_lhs ∨ val_rhs)) + (in-hole E val_new) + ∨-step + (where val_new ,(or (term val_lhs) (term val_rhs)))]))</code></pre> + +<p>The function <strong>apply-reduction-relation</strong> applies a reduction relation to a term and returns a list of ways that the term can step.</p> + +<pre><code>(apply-reduction-relation step (term #true)) +;; '() +(apply-reduction-relation step (term (#true ∧ #true))) +;; '(#true) +(apply-reduction-relation step (term (#true ∧ #false))) +;; '(#false) +(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true))) +;; '((#true ∧ #true))</code></pre> + +<p>Three things about the reduction relation <strong>step</strong>:</p> + +<ol> + <li>Using <strong>in-hole</strong> on the first argument of <strong>&mdash;&gt;</strong> searches a term for a subterm that Redex can apply a reduction rule to.</li> + <li>Using <strong>in-hole</strong> on the second argument of <strong>&mdash;&gt;</strong> puts a new value back into the <strong>hole</strong> in the evaluation context.</li> + <li>The unquote operator (<strong>,</strong>) escapes to &ldquo;Racket mode&rdquo; (see below) to evaluate a conjunction or disjunction.</li></ol> + +<p>A judgment or metafunction is a formal alternative to &ldquo;escaping to Racket&rdquo;, but escaping can be convenient.</p> + +<p>Note: the cases in a reduction relation form a <em>set</em>, not a <em>sequence</em>. If more than one case matches, Redex applies them all.</p> + +<h3 id="q-what-is-racket-mode-what-is-redex-mode">Q. What is &ldquo;Racket mode&rdquo;? What is &ldquo;Redex mode&rdquo;?</h3> + +<p>Code in a Redex model is sometimes evaluated in &ldquo;Racket mode&rdquo; and sometimes evaluated in &ldquo;Redex mode&rdquo;. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.</p> + +<p>Key points:</p> + +<ol> + <li>A Redex program starts in Racket mode.</li> + <li>The <strong>X</strong> in <strong>(term X)</strong> is evaluated in Redex mode &hellip;</li> + <li>&hellip; unless <strong>X</strong> contains unquoted sub-expressions. Unquoting escapes to Racket mode &hellip;</li> + <li>&hellip; and <strong>term</strong>s inside an unquoted sub-expression are evaluated in Redex mode.</li></ol> + +<p>In other words, <strong>term</strong> enters Redex mode and <strong>unquote</strong> (<strong>,</strong>) escapes back to Racket.</p> + +<p>Redex implicitly switches to Redex mode in a few other places, for example:</p> + +<ul> + <li>the right-side of a <strong>where</strong> clause is in Redex mode</li> + <li>the result of a metafunction is in Redex mode</li></ul> + +<p>When in doubt, try using an <strong>unquote</strong>. Redex will raise an exception if it finds an unquote in Racket mode.</p> + +<h3 id="q-are-side-conditions-evaluated-in-racket-mode-or-redex-mode">Q. Are <strong>side-condition</strong>s evaluated in &ldquo;Racket mode&rdquo; or &ldquo;Redex mode&rdquo;?</h3> + +<p>A <strong>(side-condition e)</strong> sometimes evaluates <strong>e</strong> as a Racket expression and sometimes evaluates <strong>e</strong> as a Redex expression.</p> + +<ul> + <li>reduction relations and metafunctions expect a <strong>Racket</strong> expression</li> + <li>judgments expect a <strong>Redex</strong> expression</li></ul> + +<h3 id="q-what-is-a-binding-form">Q. What is a binding form?</h3> + +<p>In the lambda calculus, <strong>λ</strong>-terms bind variables. A term <strong>(λ x M)</strong> means that any free occurrence of <strong>x</strong> in the sub-term <strong>M</strong> refers to the <strong>x</strong> from the <strong>λ</strong>-term.</p> + +<p>Redex can express this idea with a binding form.</p> + +<pre><code>(define-language Λ + [e ::= (e e) x (λ x e)] + [x ::= variable-not-otherwise-mentioned] + #:binding-forms + (λ x_0 e_0 #:refers-to x_0))</code></pre> + +<p>Note: all the non-terminals in a language must be defined before the <strong>#:binding-forms</strong> keyword. If a non-terminal definition appears after the <strong>#:binding-forms</strong> keyword, then Redex will interpret the &ldquo;definition&rdquo; as a binding form.</p> + +<p>Binding forms work together with Redex&rsquo;s functions for substitution and alphabetic equivalence.</p> + +<pre><code>(alpha-equivalent? Λ + (term (λ x x)) + (term (λ y y)))) +;; #true + +(define-metafunction Λ + test-substitute : e -&gt; e + [(test-substitute (λ x_0 e_0)) + (substitute e_0 x_0 y)]) +(term (test-substitute (λ z (z z)))) +;; '(y y)</code></pre> + +<h3 id="q-what-is--what-is-">Q. What is <strong>&hellip;</strong>? What is <strong>&hellip;.</strong>?</h3> + +<p>Three dots (<strong>&hellip;</strong>) is for building patterns. If <strong>p</strong> is a pattern then <strong>(p &hellip;)</strong> matches any list whose elements all match <strong>p</strong>.</p> + +<pre><code>(define-language L) +(redex-match? L (number ... boolean ...) (term (1 2 #true #true))) +;; #true</code></pre> + +<p>Four dots (<strong>&hellip;.</strong>) may be used in <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-extended-language%29%29"><strong>define-extended-language</strong></a> to extend a previously-defined non-terminal.</p> + +<pre><code>(define-language C + (keyword ::= auto break case)) +(define-extended-language C++ + C + (keyword ::= .... class)) + +(redex-match? C keyword (term auto)) +;; #true +(redex-match? C keyword (term class)) +;; #false +(redex-match? C++ keyword (term auto)) +;; #true +(redex-match? C++ keyword (term class)) +;; #true</code></pre> + +<h3 id="q-where-to-learn-more-about-redex">Q. Where to learn more about Redex?</h3> + +<p>&ldquo;Critical path&rdquo; resources:</p> + +<ul> + <li>Code examples for this post: <a href="https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt">https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt</a></li> + <li>Redex documentation: <a href="http://docs.racket-lang.org/redex/index.html">http://docs.racket-lang.org/redex/index.html</a></li> + <li><a href="https://dvanhorn.github.io/redex-aam-tutorial/"><em>An Introduction to Redex with Abstracting Abstract Machines</em></a> by David Van Horn</li> + <li><a href="https://www.leafac.com/playing-the-game-with-plt-redex/"><em>Playing the Game with PLT Redex</em></a> by Leandro Facchinetti</li> + <li><a href="https://williamjbowman.com/doc/experimenting-with-redex/index.html"><em>Experimenting with Languages in Redex</em></a> by William J. Bowman</li></ul> + +<p>&ldquo;Procrastination&rdquo; resources:</p> + +<ul> + <li><a href="http://tata.gforge.inria.fr/"><em>Tree Automata Techniques and Applications</em></a></li> + <li><a href="http://lamport.azurewebsites.net/pubs/lamport-types.pdf"><em>Should your Specification Language be Typed?</em></a></li> + <li>Redex source code (see <code>redex-lib/</code>): <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + + PLT Redex: mf-apply + http://prl.ccs.neu.edu/blog/2017/03/03/plt-redex-mf-apply/?utm_source=PLT-Redex&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-03-03-plt-redex-mf-apply + Fri, 03 Mar 2017 08:54:20 UT + Ben Greenman + +<p>The <code>mf-apply</code> keyword is for checked metafunction application in PLT Redex. In other words, <code>(mf-apply f x)</code> is just like <code>(f x)</code>, but errors if <code>f</code> is not a previously-defined metafunction.</p> + +<p>Also, consider applying to attend <em>The Racket School of Semantics and Languages</em> in Salt Lake City this summer: <a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></p> +<!-- more--> + +<h2 id="metafunctions-vs-list-patterns">Metafunctions vs. List Patterns</h2> + +<p>Have you used PLT Redex? Good! Maybe this has happened to you:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span> +<span class="normal">25</span> +<span class="normal">26</span> +<span class="normal">27</span> +<span class="normal">28</span> +<span class="normal">29</span> +<span class="normal">30</span> +<span class="normal">31</span> +<span class="normal">32</span> +<span class="normal">33</span> +<span class="normal">34</span> +<span class="normal">35</span> +<span class="normal">36</span> +<span class="normal">37</span> +<span class="normal">38</span> +<span class="normal">39</span> +<span class="normal">40</span> +<span class="normal">41</span> +<span class="normal">42</span> +<span class="normal">43</span> +<span class="normal">44</span> +<span class="normal">45</span> +<span class="normal">46</span> +<span class="normal">47</span> +<span class="normal">48</span> +<span class="normal">49</span> +<span class="normal">50</span> +<span class="normal">51</span> +<span class="normal">52</span> +<span class="normal">53</span> +<span class="normal">54</span> +<span class="normal">55</span> +<span class="normal">56</span> +<span class="normal">57</span> +<span class="normal">58</span> +<span class="normal">59</span> +<span class="normal">60</span> +<span class="normal">61</span> +<span class="normal">62</span> +<span class="normal">63</span> +<span class="normal">64</span> +<span class="normal">65</span> +<span class="normal">66</span> +<span class="normal">67</span> +<span class="normal">68</span> +<span class="normal">69</span> +<span class="normal">70</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 1. You define a language</span> +<span class="p">(</span><span class="n">define-language</span> <span class="n">STLC</span> + <span class="p">[</span><span class="n">V</span> <span class="n">::=</span> <span class="n">integer</span> <span class="n">boolean</span> <span class="n">C</span><span class="p">]</span> + <span class="p">[</span><span class="n">C</span> <span class="n">::=</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">Λ</span> <span class="n">::=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)]</span> + <span class="p">[</span><span class="n">M</span> <span class="n">::=</span> <span class="p">(</span><span class="n">M</span> <span class="n">M</span><span class="p">)</span> <span class="n">V</span> <span class="n">Λ</span> <span class="n">x</span><span class="p">]</span> + <span class="p">[</span><span class="n">τ</span> <span class="n">::=</span> <span class="n">Int</span> <span class="n">Bool</span> <span class="p">(</span><span class="n">τ</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">ρ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">V</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">Γ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">τ</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">x</span> <span class="n">::=</span> <span class="n">variable-not-otherwise-mentioned</span><span class="p">]</span> + <span class="kd">#:binding-forms</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span> <span class="kd">#:refers-to</span> <span class="n">x</span><span class="p">))</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 2. You define a few metafunctions</span> +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;lam</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">Λ</span> + <span class="p">[(</span><span class="n">closure-&gt;lam</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">Λ</span><span class="p">])</span> + +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;env</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">ρ</span> + <span class="p">[(</span><span class="n">closure-&gt;env</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">ρ</span><span class="p">])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 3. You try defining a judgment form . . .</span> +<span class="p">(</span><span class="n">define-judgment-form</span> <span class="n">STLC</span> + <span class="kd">#:mode</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">I</span> <span class="n">O</span><span class="p">)</span> + <span class="kd">#:contract</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Var</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">x</span> <span class="p">(</span><span class="n">x</span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_0</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_1</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">---</span> <span class="n">FVS-App</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">M_0</span> <span class="n">M_1</span><span class="p">)</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x_0</span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)</span> <span class="n">Λ</span><span class="p">)</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-remove))" style="color: inherit">set-remove</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="n">x_0</span><span class="p">)))</span> + <span class="n">---</span> <span class="n">FVS-Λ</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Integer</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">integer_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Boolean</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">boolean_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">closure-&gt;lam</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-subtract))" style="color: inherit">set-subtract</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))))</span> + <span class="n">---</span> <span class="n">FVS-Closure</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">C</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 4. You test the judgment, and it mysteriously fails</span> +<span class="p">(</span><span class="n">judgment-holds</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">closure</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">())</span> + <span class="p">()))</span> +<span class="c1">;; ==&gt; #f</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><strong>WHAT HAPPENED??!</strong></p> + +<p>The problem is this line in the <code>FVS-Closure</code> rule:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which checks that the list <code>(closure-env C)</code> (whose first element is the symbol <code>closure-env</code> and second element is the symbol <code>C</code>) matches the pattern <code>((x_1 τ_1) ...)</code>.</p> + +<p>Right.</p> + +<p>Of course you meant to apply the metafunction <code>closure-&gt;env</code> but made a typo. And since the syntax for metafunction application is the same as the syntax for building a list, Redex doesn&rsquo;t report an error.</p> + +<p>We can fix this code with the new <a href="https://www.cs.utah.edu/plt/snapshots/current/doc/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._mf-apply%29%29"><code>mf-apply</code></a> keyword (available on <a href="https://github.com/racket/racket">GitHub</a> or in a <a href="https://www.cs.utah.edu/plt/snapshots/">snapshot build</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running <code>raco make</code> now gives a compile-time error.</p> + +<pre><code> term: expected a previously defined metafunction + at: closure-env + in: (mf-apply closure-env C)</code></pre> + +<h3 id="but-i-still-need-to-type-mf-apply-correctly">But I still need to type <code>mf-apply</code> correctly!</h3> + +<p>Leif Andersen says:</p> + +<blockquote> + <p>I should point out that this has the issue of you still need to type <code>mf-apply</code> correctly. ;)</p></blockquote> + +<p>That is, if you accidentally write:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-applu</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then the code compiles, thinking you intend to match a list of three elements against the pattern.</p> + +<p>Never fear, there are at least two solutions.</p> + +<h4 id="solution-1-rename-mf-apply">Solution 1: rename <code>mf-apply</code></h4> + +<p>A simple fix is to rename the <code>mf-apply</code> keyword to something shorter (and harder to mis-type):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._rename-in))" style="color: inherit">rename-in</a></span> <span class="n">redex</span> + <span class="p">[</span><span class="n">mf-apply</span> <span class="n">MF</span><span class="p">]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="solution-2-the-mf-apply-lang-extension">Solution 2: the <code>mf-apply</code> lang extension</h4> + +<p>A fancier solution is to install the <code>mf-apply</code> meta-language.</p> + +<pre><code> $ raco pkg install mf-apply</code></pre> + +<p>This language updates the <a href="http://docs.racket-lang.org/reference/readtables.html#%28tech._readtable%29"><em>readtable</em></a> to interpret S-expressions that begin with <code>#{</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="o">#</span><span class="p">{</span><span class="ss">f</span> <span class="ss">x</span> <span class="ss"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>as a metafunction application:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">f</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>You the programmer only needs to write the <code>#{....}</code> syntax.</p> + +<p>Source code is on GitHub:</p> + +<ul> + <li><a href="https://github.com/bennn/mf-apply">https://github.com/bennn/mf-apply</a></li></ul> + +<p>(It&rsquo;s the simplest lang-extension I know of)</p> + +<h2 id="what-is-plt-redex">What is PLT Redex?</h2> + +<p>PLT Redex is a library for semantics engineering. One of my favorite Redex features is it implements capture-avoiding substitution and α-equivalence for any language with a <code>#:binding-forms</code> specification (such as STLC, above).</p> + +<p>You can read more:</p> + +<ul> + <li>in the &ldquo;Amb&rdquo; tutorial: <a href="http://docs.racket-lang.org/redex/tutorial.html">http://docs.racket-lang.org/redex/tutorial.html</a></li> + <li>in the &ldquo;Long Tutorial&rdquo;: <a href="http://docs.racket-lang.org/redex/redex2015.html">http://docs.racket-lang.org/redex/redex2015.html</a></li> + <li>in the Redex reference manual: <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html">http://docs.racket-lang.org/redex/The_Redex_Reference.html</a></li> + <li>on the PLT Redex website: <a href="https://redex.racket-lang.org/">https://redex.racket-lang.org/</a></li> + <li>on GitHub: <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + +<p>And if you act now, you can become a <em>Redexan</em> between July 10 and July 14 at the summer school in Salt Lake City, Utah:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Racket.atom.xml b/blog/feeds/Racket.atom.xml new file mode 100644 index 00000000..9f463680 --- /dev/null +++ b/blog/feeds/Racket.atom.xml @@ -0,0 +1,3322 @@ + + + PRL Blog: Posts tagged 'Racket' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Racket-html + 2018-04-27T21:35:22Z + + The Racket School 2018: Create your own language + + urn:http-prl-ccs-neu-edu:-blog-2018-04-27-the-racket-school-2018-create-your-own-language + 2018-04-27T21:35:22Z + 2018-04-27T21:35:22Z + + Ben Greenman + +<p>The Racket School 2018: Create your own language • 9–13 July • Salt Lake City</p> +<!-- more--> + +<p>The Racket team has spent over thirty years developing and refining a coherent intellectual tradition for studying and building programming languages. This year’s school will introduce participants to Racket’s framework for language-oriented programming, which the summer school faculty recently spelled out in a a cover article in the <a href="https://tinyurl.com/RacketCACM">Communications of the ACM</a></p> + +<p>Concretely, the 2018 Racket Summer School will cover the following topics:</p> + +<ul> + <li>the spectrum of programming languages;</li> + <li>modules and syntax, or languages as libraries;</li> + <li>DrRacket’s support for language-oriented programming;</li> + <li>a domain-specific language for adding types to languages;</li> + <li>tools and techniques for implementing notational conveniences; and</li> + <li>research challenges in language-oriented programming.</li></ul> + +<p>If these topics intrigue you, attend the Racket Summer School:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2018/">http://summer-school.racket-lang.org/2018/</a></li></ul> + +<p>This is not your run-of-the-mill summer school. We will do our best to make it exciting, entertaining, and useful to a broad spectrum of attendees, both academic and industrial.</p> + +<p>P.S. We will send you your first problem set in June, a month before the summer school to whet your appetite.</p> + + Tutorial: Zero to Sixty in Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-08-02-tutorial-zero-to-sixty-in-racket + 2016-08-02T01:29:11Z + 2016-08-02T01:29:11Z + + Ben Greenman + +<p>Racket is excellent for incrementally growing scripts into full-fledged programs. +This post steps through the evolution of one small program and highlights the + Racket tools that enable incremental advances.</p> +<!--more--> + +<p></p> + +<div class="SIntrapara">Why should anyone use <a href="http://racket-lang.org/">Racket</a>? +There are two reasons: +</div> + +<div class="SIntrapara"> + <ol> + <li> + <p>You have a problem that can only be solved with Racket&rsquo;s language-building tools</p></li> + <li> + <p>Racket is a nice language to program in. +(Has lexical scope, parentheses, <a href="http://con.racket-lang.org">active users</a>...)</p></li></ol></div> + +<p>My favorite part of Racket is how it supports a certain development style of + evolving scripts into programs. +When I start coding (after design, before debugging), I can focus the problem at hand. +Next come examples, unit tests, and types to be sure the solution is correct. +Finally, I worry about aesthetics, efficiency, and how the solution can be used as a library in a larger context.</p> + +<p>Bottom line: with Racket, my coding is aligned with my priorities. +And as I transition from "no code" to "working code" to "robust code" to "re-usable code", + the program is almost always runnable.</p> + +<h1><a name="(part._.Problem__.A_.K.W.I.C_.Index_.Production_.System)"></a>Problem: A KWIC Index Production System</h1> + +<p>A KWIC index system +reads input from a file, +divides each line of the file into whitespace-separated words, +and outputs (in alphabetical order) all circular shifts of all lines.</p> + +<p>The first circular shift of a line <span class="RktWrap"><span class="RktVal">"A B C"</span></span> is the line <span class="RktWrap"><span class="RktVal">"B C A"</span></span>. +The second circular shift is <span class="RktWrap"><span class="RktVal">"C A B"</span></span>.</p> + +<p>Building a KWIC index is a historical problem. +According to <a href="https://www.cs.umd.edu/class/spring2003/cmsc838p/Design/criteria.pdf">D.L. Parnas (1972)</a>:</p> + +<blockquote class="SubFlow"> + <p>Except under extreme circumstances (huge data base, no supporting software) +such a system could be implemented by a good programmer within a week or two.</p></blockquote> + +<p>See also: <a href="https://yanniss.github.io/law.html">Yannis&rsquo;s Law</a>.</p> + +<p>Today, I bet only <a href="http://wiki.portal.chalmers.se/agda/pmwiki.php">Agda</a> and <a href="https://scratch.mit.edu/">Scratch</a> + programmers would need the full two weeks. +We&rsquo;ll be done in 20 minutes.</p> + +<h1><a name="(part._.A_.Script)"></a>A Script</h1> + +<p>To start, open a file and type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>You can name the file anything, like <span class="stt">kwic.rkt</span> or <span class="stt">rkt.kwic</span> or <span class="stt">foo</span>. +Racket doesn&rsquo;t care, + but it does need the <span class="stt">#lang</span> line to read the contents of the file.</p> + +<p>Though, you should use the <span class="stt">.rkt</span> extension.</p> + +<p>The first part of the solution is a function to read input from a file into a + list of strings for further processing. +The built-in function <a href="http://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28lib._racket%2Ffile..rkt%29._file-~3elines%29%29">file-&gt;lines</a> + does exactly this, but we&rsquo;ll for-loop instead.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="hspace">&nbsp;</span><span class="RktSym">filename</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="hspace">&nbsp;</span><span class="RktSym">filename</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>When called with a filename like <span class="RktWrap"><span class="RktVal">"heart-of-darkness.txt"</span></span>, the function + uses <a href="http://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a> to build a list of lines by reading from a port + with <a href="http://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a>. +The port is the data from <span class="RktWrap"><span class="RktSym">filename</span></span>, thanks to <a href="http://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a>.</p> + +<p>Next is a function to convert a list of strings into a list of lists of words. +Here we&rsquo;ll just use library functions.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktSym">lines</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="hspace">&nbsp;</span><span class="RktSym">string-split</span><span class="hspace">&nbsp;</span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>By default, <a href="http://docs.racket-lang.org/reference/strings.html#%28def._%28%28lib._racket%2Fstring..rkt%29._string-split%29%29">string-split</a> divides a string into a list of whitespace-separated substrings. +You can always supply a different delimiter, or use <a href="http://docs.racket-lang.org/reference/regexp.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._regexp-split%29%29">regexp-split</a> + to divide by a regular expression.</p> + +<p>Two tasks left! +First we generate all circular shifts for a list of strings <span class="RktWrap"><span class="RktSym">words</span></span> + by folding up a list with one shift of <span class="RktWrap"><span class="RktSym">words</span></span> for each word.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Second, we alphabetize and print the shifts.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">destruct multiple values</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">first list empty, don</span><span class="RktCmt">'</span><span class="RktCmt">t care about second</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#t</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">first list non-empty, second empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1-rest</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="hspace">&nbsp;</span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktVal">" "</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">for-each is like map, but returns (void)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="hspace">&nbsp;</span><span class="RktSym">display-words</span><span class="hspace">&nbsp;</span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Gluing it all together, here&rsquo;s the full script (with type annotations in comments).</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">type</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">=</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">type</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">=</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Path-String</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Move</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">element</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">last</span><span class="hspace">&nbsp;</span><span class="RktCmt">position</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lexicographic</span><span class="hspace">&nbsp;</span><span class="RktCmt">order</span><span class="hspace">&nbsp;</span><span class="RktCmt">on</span><span class="hspace">&nbsp;</span><span class="RktCmt">equal-length</span><span class="hspace">&nbsp;</span><span class="RktCmt">lists</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Boolean</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">don't</span><span class="hspace">&nbsp;</span><span class="RktCmt">care</span><span class="hspace">&nbsp;</span><span class="RktCmt">about</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">non-empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/booleans.html#%28def._%28%28quote._~23~25kernel%29._not%29%29">not</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._null~3f%29%29">null?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"</span><span class="hspace">&nbsp;</span><span class="RktVal">"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Path-String</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">append*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">End-to-end</span><span class="hspace">&nbsp;</span><span class="RktCmt">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">run-test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"test.txt"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Make</span><span class="hspace">&nbsp;</span><span class="RktCmt">a</span><span class="hspace">&nbsp;</span><span class="RktCmt">file</span><span class="hspace">&nbsp;</span><span class="RktCmt">and</span><span class="hspace">&nbsp;</span><span class="RktCmt">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/when_unless.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._unless%29%29">unless</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._file-exists~3f%29%29">file-exists?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">run-test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Running the file should print:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<h1><a name="(part._.Testing_and_.Submodules)"></a>Testing and Submodules</h1> + +<p>Any top-level expressions in a file can work as unit tests. +The <a href="http://docs.racket-lang.org/reference/booleans.html%3F#%28def._%28%28quote._~23~25kernel%29._equal~3f%29%29">equal?</a> statement below checks whether the first circular shift + of <span class="RktWrap"><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="stt"> </span><span class="RktVal">"B"</span><span class="stt"> </span><span class="RktVal">"C"</span><span class="RktVal">)</span></span> is <span class="RktWrap"><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="stt"> </span><span class="RktVal">"C"</span><span class="stt"> </span><span class="RktVal">"A"</span><span class="RktVal">)</span></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Equality.html#%28def._%28%28quote._~23~25kernel%29._equal~3f%29%29">equal?</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Running the file now prints <span class="RktWrap"><span class="RktVal">#t</span></span> to the console, meaning the test passed. +We can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/exns.html#%28def._%28%28quote._~23~25kernel%29._error%29%29">error</a></span></span> or <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/exns.html#%28def._%28%28quote._~23~25kernel%29._raise-user-error%29%29">raise-user-error</a></span></span> to make failures easier + to notice. +Or we can use the <a href="http://docs.racket-lang.org/rackunit/api.html">RackUnit</a> testing library.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">import the testing library</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>These tests run each time the module does. +If you prefer to run tests only in a specific context, and not when the + module is run or imported as a library, you can move them to a separate + file or into a <a href="http://docs.racket-lang.org/reference/eval-model.html#%28tech._submodule%29">submodule</a>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Open a submodule named </span><span class="RktCmt">'</span><span class="RktCmt">test</span><span class="RktCmt">'</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Running the module normally via <span class="stt">racket kwic.rkt</span> will not run code + in the submodule. +Instead, use <span class="stt">raco test</span> to run the tests.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">raco</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">raco</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._submod%29%29">submod</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"kwic.rkt"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">passed</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The reason we used <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span></span>, instead of Racket&rsquo;s <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%29%29">module</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%2A%29%29">module*</a></span></span> + forms is that <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span></span> inherits the language and namespace of its + containing module and can be incrementally extended. +This way, we can keep tests near the relevant code.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"is"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"as"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"does"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"as"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"does"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"is"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p><a href="http://docs.racket-lang.org/rackunit/api.html">RackUnit</a> in a + separate file or <span class="RktWrap"><span class="RktSym">test</span></span> submodule is the unofficial standard for testing + Racket programs.</p> + +<h1><a name="(part._.Recognizing_.Patterns__.Avoiding_.Repetition)"></a>Recognizing Patterns, Avoiding Repetition</h1> + +<p>Every unit test we&rsquo;ve written uses <a href="http://docs.racket-lang.org/rackunit/api.html#%28def._%28%28lib._rackunit%2Fmain..rkt%29._check-equal~3f%29%29">check-equal?</a>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"hello"</span><span class="hspace">&nbsp;</span><span class="RktVal">"world"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">" lost "</span><span class="hspace">&nbsp;</span><span class="RktVal">" in "</span><span class="hspace">&nbsp;</span><span class="RktVal">"space"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"lost"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"in"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"space"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"something"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>These tests follow a simple pattern that we can express as a <span class="emph">syntax rule</span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._define-syntax-rule%29%29">define-syntax-rule</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="hspace">&nbsp;</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="RktPn">]</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/begin.html#%28form._%28%28quote._~23~25kernel%29._begin%29%29">begin</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"hello"</span><span class="hspace">&nbsp;</span><span class="RktVal">"world"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">" out "</span><span class="hspace">&nbsp;</span><span class="RktVal">" in "</span><span class="hspace">&nbsp;</span><span class="RktVal">"the ozone"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"out"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"in"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"the"</span><span class="hspace">&nbsp;</span><span class="RktVal">"ozone"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"something"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span></span> are not pseudocode! +They denote Kleene-star repetition, like a sextile (<span class="RktWrap"><span class="RktVal">"*"</span></span>) in a regular expression. +In this case, the input pattern is a sequence of lists with two S-expressions, <span class="RktWrap"><span class="RktSym">i</span></span> and + <span class="RktWrap"><span class="RktSym">o</span></span>. +Uses of <span class="RktWrap"><span class="RktSym">i</span></span> and <span class="RktWrap"><span class="RktSym">o</span></span> in the rule must be followed by one <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span></span> to splice + the captured S-expressions into the result.</p> + +<p>Many languages offer higher-order functions and polymorphism to abstract common + behaviors. +Syntax extensions are a different way to avoid repeating yourself. +After 30 years, we are still discovering what syntax extensions are useful for.</p> + +<p>See this <a href="https://groups.google.com/forum/#!topic/racket-users/ss20lwfUhjs/discussion">recent Racket mailing list post</a> for some applications.</p> + +<h1><a name="(part._.Adding_.Static_.Types)"></a>Adding Static Types</h1> + +<p>Changing the <a href="http://docs.racket-lang.org/reference/reader.html#%28idx._%28gentag._79._%28lib._scribblings%2Freference%2Freference..scrbl%29%29%29"><span class="stt">#lang</span></a> line to <span class="RktWrap"><span class="RktSym">typed/racket</span></span> adds static type-checking to our program. +If we only change the language and run the code as-is, there will be type errors. +But we can use submodules again to incrementally check our design with types.</p> + +<p>Note: <a href="http://docs.racket-lang.org/ts-reference/Typed_Regions.html">typed regions</a> + are another way to embed typed code into untyped contexts.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%29%29">module</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">typed/racket</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Need</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">annotate:</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-</span><span class="hspace">&nbsp;</span><span class="RktCmt">function</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameters</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-</span><span class="hspace">&nbsp;</span><span class="RktCmt">for-loop</span><span class="hspace">&nbsp;</span><span class="RktCmt">return</span><span class="hspace">&nbsp;</span><span class="RktCmt">types</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Next</span><span class="hspace">&nbsp;</span><span class="RktCmt">migration</span><span class="hspace">&nbsp;</span><span class="RktCmt">step:</span><span class="hspace">&nbsp;</span><span class="RktCmt">move</span><span class="hspace">&nbsp;</span><span class="RktCmt">other</span><span class="hspace">&nbsp;</span><span class="RktCmt">untyped</span><span class="hspace">&nbsp;</span><span class="RktCmt">functions</span><span class="hspace">&nbsp;</span><span class="RktCmt">here</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._all-defined-out%29%29">all-defined-out</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">t</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">&lt;rest</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">file</span><span class="hspace">&nbsp;</span><span class="RktCmt">omitted&gt;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>After scooping all functions into the Typed Racket bubble, we can remove the + submodule declaration and change <span class="stt">#lang racket</span> to <span class="stt">#lang typed/racket</span>.</p> + +<h1><a name="(part._.Finally__a_.Library)"></a>Finally, a Library</h1> + +<p>Other modules can import our functions if we use a <a href="http://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a> statement. +By <a href="https://docs.racket-lang.org/style/Units_of_Code.html">convention</a>, exports belong at the top of a file.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">#lang</span><span class="hspace">&nbsp;</span><span class="RktMeta">typed/racket</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-index</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">&lt;definitions</span><span class="hspace">&nbsp;</span><span class="RktCmt">here&gt;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Then any typed or untyped module can use <span class="RktWrap"><span class="RktSym">kwic-index</span></span> by writing + <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="stt"> </span><span class="RktVal">"kwic.rkt"</span><span class="RktPn">)</span></span>.</p> + +<p>As a finishing touch, we can use the <a href="http://docs.racket-lang.org/reference/Command-Line_Parsing.html">racket/cmdline</a> library + inside a <span class="stt">main</span> submodule to give a basic front-end interface. +Similar to <span class="stt">module+ test</span>, a <span class="stt">module+ main</span> declares code that + inherits the file&rsquo;s bindings and language but is only run when the program + is executaed.</p> + +<p>Here is the complete typed and tested code listing. +The <span class="RktWrap"><span class="RktSym">main</span></span> submodule is at the bottom.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/ts-reference/index.html"><span class="RktSym">typed/racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">typed/rackunit</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._define-syntax-rule%29%29">define-syntax-rule</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">o</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/begin.html#%28form._%28%28quote._~23~25kernel%29._begin%29%29">begin</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">o</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">define-type</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">define-type</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">make-temporary-file</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"The</span><span class="hspace">&nbsp;</span><span class="RktVal">Nellie,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"a</span><span class="hspace">&nbsp;</span><span class="RktVal">cruising</span><span class="hspace">&nbsp;</span><span class="RktVal">yawl,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"swung</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">her</span><span class="hspace">&nbsp;</span><span class="RktVal">anchor</span><span class="hspace">&nbsp;</span><span class="RktVal">without</span><span class="hspace">&nbsp;</span><span class="RktVal">a</span><span class="hspace">&nbsp;</span><span class="RktVal">flutter</span><span class="hspace">&nbsp;</span><span class="RktVal">of</span><span class="hspace">&nbsp;</span><span class="RktVal">sails,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"and</span><span class="hspace">&nbsp;</span><span class="RktVal">was</span><span class="hspace">&nbsp;</span><span class="RktVal">at</span><span class="hspace">&nbsp;</span><span class="RktVal">rest."</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">actual</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expect</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">file-&gt;lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._delete-file%29%29">delete-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">actual</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expect</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#{</span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta">::</span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"hello"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"world"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Move</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">element</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">last</span><span class="hspace">&nbsp;</span><span class="RktCmt">position</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lexicographic</span><span class="hspace">&nbsp;</span><span class="RktCmt">order</span><span class="hspace">&nbsp;</span><span class="RktCmt">on</span><span class="hspace">&nbsp;</span><span class="RktCmt">equal-length</span><span class="hspace">&nbsp;</span><span class="RktCmt">lists</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Boolean</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">don't</span><span class="hspace">&nbsp;</span><span class="RktCmt">care</span><span class="hspace">&nbsp;</span><span class="RktCmt">about</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">non-empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"</span><span class="hspace">&nbsp;</span><span class="RktVal">"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._open-output-string%29%29">open-output-string</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._get-output-string%29%29">get-output-string</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A\nB</span><span class="hspace">&nbsp;</span><span class="RktVal">C\n"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"D"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"D"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">append*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._open-output-string%29%29">open-output-string</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">make-temporary-file</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._delete-file%29%29">delete-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._get-output-string%29%29">get-output-string</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">string-join</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"if</span><span class="hspace">&nbsp;</span><span class="RktVal">this</span><span class="hspace">&nbsp;</span><span class="RktVal">imagine"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"this</span><span class="hspace">&nbsp;</span><span class="RktVal">imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to\n"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"\n"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">main</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket/cmdline</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Parameterof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Any</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28def._%28%28quote._~23~25kernel%29._make-parameter%29%29">make-parameter</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">command-line</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:program</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"kwic</span><span class="hspace">&nbsp;</span><span class="RktVal">index"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:once-each</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktVal">"-o"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"--output"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">user-supplied</span><span class="hspace">&nbsp;</span><span class="RktCmt">input</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"Write</span><span class="hspace">&nbsp;</span><span class="RktVal">output</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">file"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">update</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameter</span><span class="hspace">&nbsp;</span><span class="RktCmt">*output-to*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:args</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">*output-to*</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">read</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameter</span><span class="hspace">&nbsp;</span><span class="RktCmt">*output-to*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3f%29%29">string?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._open-output-file%29%29">open-output-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">cast</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/when_unless.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._when%29%29">when</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3f%29%29">string?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._close-output-port%29%29">close-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">Sample interactions: +</div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">kwic</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expects</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">&lt;file-name&gt;</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">on</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">the</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">command</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quasiquote.html#%28form._%28%28quote._~23~25kernel%29._unquote%29%29">,</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">given</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">arguments</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">echo</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"It</span><span class="hspace">&nbsp;</span><span class="RktVal">is</span><span class="hspace">&nbsp;</span><span class="RktVal">a</span><span class="hspace">&nbsp;</span><span class="RktVal">truth</span><span class="hspace">&nbsp;</span><span class="RktVal">universally</span><span class="hspace">&nbsp;</span><span class="RktVal">acknowledged"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">pride-and-prejudice.txt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-o</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">pride-and-prejudice.txt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">wc</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-l</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktVal">6</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span></td></tr></tbody></table></div></div> + +<h1><a name="(part._.Closing)"></a>Closing</h1> + +<p>We started with functions, wrote (and quarantined) unit tests, + reinforced our design with types, and added a command-line interface. +Going forward we could add <a href="http://docs.racket-lang.org/scribble/index.html">Scribble</a> documentation and share our work as a <a href="http://pkgn.racket-lang.org/">package</a>.</p> + +<p></p> + +<div class="SIntrapara">For more on building languages with Racket: +</div> + +<div class="SIntrapara"> + <ul> + <li> + <p><a href="http://www.hashcollision.org/brainfudge/">Fudging up a Racket (html)</a></p></li> + <li> + <p><a href="http://dl.acm.org/authorize?6529547">Creating Languages in Racket (pdf)</a></p></li> + <li> + <p><a href="http://www.ccs.neu.edu/home/matthias/manifesto/">The Racket Manifesto (html)</a></p></li> + <li> + <p><a href="http://www.terohasu.net/hasu-flatt--els16--preprint.pdf">Source-to-Source Compilation via Submodules (pdf)</a></p></li></ul></div> + + Tutorial: Racket FFI, part 3 + + urn:http-prl-ccs-neu-edu:-blog-2016-07-11-tutorial-racket-ffi-part-3 + 2016-07-11T17:33:40Z + 2016-07-11T17:33:40Z + + Asumu Takikawa + +<p>This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a> +and part 2 +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a>.</p> + +<p>In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.</p> +<!--more--> + +<p>As in the second post, let&rsquo;s start with some prologue code that establishes +the definitions from the previous two posts. But first, I&rsquo;m getting tired of +writing the <span class="stt">#:c-id identifier</span> notation for the underscored C function +names.</p> + +<p>Instead, let&rsquo;s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">$</span><span class="hspace">&nbsp;</span><span class="RktMeta">raco</span><span class="hspace">&nbsp;</span><span class="RktMeta">pkg</span><span class="hspace">&nbsp;</span><span class="RktMeta">install</span><span class="hspace">&nbsp;</span><span class="RktMeta">ffi-definer-convention</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>or you can just execute the following snippet in Racket:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pkg</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">pkg-install-command</span><span class="hspace">&nbsp;</span><span class="RktPn">#:skip-installed</span><span class="hspace">&nbsp;</span><span class="RktVal">#t</span><span class="hspace">&nbsp;</span><span class="RktVal">"ffi-definer-convention"</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Downloading repository git://github.com/takikawa/racket-ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: version: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: platform: x86_64-linux [3m]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: target machine: racket</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: installation name: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: variants: 3m</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main collects: /usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: collects paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main pkgs: /usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: pkgs paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: links files: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main docs: /usr/share/racket/doc</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- updating info-domain tables ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- pre-installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing foreign libraries --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing shared files ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- compiling collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- parallel build using 4 jobs ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 making: &lt;pkgs&gt;/ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- creating launchers --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing man pages --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- building documentation --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 running: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 rendering: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 2 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/local-redirect.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 1 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/release.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/search.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/start.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- post-installing collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This will install the package and compile its contents. If you&rsquo;re curious, the +docs for the package are available +<a href="http://docs.racket-lang.org/ffi-definer-convention/index.html">here</a>.</p> + +<p><span style="font-weight: bold">Note:</span> if you&rsquo;ve never installed a package before, you may want to glance +at the +<a href="http://docs.racket-lang.org/pkg/getting-started.html">package system docs</a>. +A tl;dr of packages is that they bundle Racket <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/collects.html#%28tech._collection%29"><span class="techinside">collections</span></a>, which are +sets of modules that you can refer to in a location-independent fashion such as +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> or <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28mod-path._racket%2Flist%29"><span class="RktSym">racket/list</span></a></span>.</p> + +<p>Anyhow, here&rsquo;s the prologue code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">avoid conflict with below</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._except-in%29%29">except-in</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">define-ffi-definer</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the new 3rd-party pkg</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi-definer-convention</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">describes how to transform from</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Racket to C ids</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:make-c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">convention:hyphen-&gt;underscore</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">note lack of #:c-id keyword arguments</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">(_cairo_t -&gt; Void) -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">do some drawing and give us the pict</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">f</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">f</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Notice that the <span class="RktWrap"><span class="RktSym">define-cairo</span></span> forms don&rsquo;t have any <span class="stt">#:c-id</span> keywords +anymore. Instead, the prologue code uses an overriden <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> +from my package that supports a <span class="stt">#:make-c-id</span> keyword that lets you specify +a naming convention to follow.</p> + +<p>Also, instead of creating a single bitmap and drawing into it, we now have a +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> function that takes a drawing function. When called, +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> will call the given function with a new bitmap object and +return the result.</p> + +<p>Now let&rsquo;s get to the main point of this blog post. Let&rsquo;s say that we want to play +with Cairo <a href="https://www.cairographics.org/manual/cairo-Paths.html">path</a> +objects this time. A path is +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-t">defined</a> +as a struct with the following structure:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_status_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">status</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it&rsquo;s +useful to define C types for the types of values in the path struct&rsquo;s fields. First, +let&rsquo;s specify that a <span class="stt">cairo_status_t</span> is an integer type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>It&rsquo;s actually an enum, but for the examples in this post we don&rsquo;t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-data-t">path data objects</a>. +Each path data object is a <span class="stt">cairo_path_data_t</span>, +which is specified with a C union:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">union</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">_cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_type_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">type</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">length</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">header</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">point</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Helpfully, the FFI library comes with support for unions with the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span></span> type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It&rsquo;s pretty +straightforward to specify this type too:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the path data type is just an enum</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">move-to</span><span class="hspace">&nbsp;</span><span class="RktVal">line-to</span><span class="hspace">&nbsp;</span><span class="RktVal">curve-to</span><span class="hspace">&nbsp;</span><span class="RktVal">close-path</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the header case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the point case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>There&rsquo;s a new type constructor here so let me explain that first. +The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span></span> constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span>, this constructor doesn&rsquo;t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.</p> + +<p>Each of the path data structs in the path data array will be manipulated with +the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">create a union from a list of doubles</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1.3</span><span class="hspace">&nbsp;</span><span class="RktVal">5.8</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">source type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">target type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;union&gt;</span></p></td></tr></tbody></table></div> + +<p>This snippet first construct a union object (via the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> +type) using a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span></span>. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it&rsquo;s an easy way +to generate a union object.</p> + +<p>The second line shows that a union prints as an opaque object. You can&rsquo;t do +anything with a union in Racket unless you project it to one of the sub-cases with +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></span> function. +This projection is <span class="emph">unsafe</span>, in the sense that if you don&rsquo;t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.</p> + +<p>More concretely, let&rsquo;s see what happens if we try to extract a value out of the +union both correctly and incorrectly:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">correct (matches construction)</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">cases are zero-indexed and ordered as written</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(1.3 5.8)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">incorrect, error</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">enum:int-&gt;_cairo_path_data_type_t: expected a known</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">#&lt;ctype:ufixint&gt;, got: 3435973837</span></p></td></tr></tbody></table></div> + +<p>Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.</p> + +<p>With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.</p> + +<p>With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it&rsquo;s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.</p> + +<p>So before moving on, let&rsquo;s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we&rsquo;ll figure out how to +deal with the array itself.</p> + +<h1><a name="(part._.Some_low-level_operations)"></a>Some low-level operations</h1> + +<p>Since we still don&rsquo;t have a C type for <span class="stt">cairo_path_t</span>, let&rsquo;s go ahead +and make a simple one where we punt on the work of specifying the array +type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In this type, we have specified the array as a bare <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span>. +For some added safety, we could also use something like +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__cpointer%29%29">_cpointer</a></span><span class="stt"> </span><span class="RktVal">'</span><span class="RktVal">cairo_status_t</span><span class="RktPn">)</span></span>, which sets up a tagged pointer +type like we saw in the first blog post with +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>.</p> + +<p>We&rsquo;ve seen the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type before, but haven&rsquo;t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.</p> + +<p>Before we get to that, let&rsquo;s go ahead and set up an FFI binding for +<span class="stt">cairo_copy_path</span> so that we can obtain a path struct to manipulate:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Do stuff to make the current</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">path non-empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Get the current path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/set_.html#%28form._%28%28quote._~23~25kernel%29._set%21%29%29">set!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Stroke clears the path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">so do it last</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<p>Note that <span class="RktWrap"><span class="RktSym">cairo-copy-path</span></span> gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span>, which +lets you dereference a pointer and access it at some concrete C type.</p> + +<p><span style="font-weight: bold">Note:</span> the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> function also takes an optional +offset argument which we will be used in an example later.</p> + +<p>For example, we can use <span class="RktWrap"><span class="RktSym">a-path</span></span> as a <span class="RktWrap"><span class="RktSym">_simple_cairo_path_t</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(0 #&lt;cpointer&gt; 8)</span></p></td></tr></tbody></table></div> + +<p>And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> again with an array type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">array</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">length field</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">array</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(#&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt;)</span></p></td></tr></tbody></table></div> + +<p>The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(move-to 2)</span></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(50.0 50.0)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">nonsense data here, wrong union case</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(4.2439915824246e-314 0.0)</span></p></td></tr></tbody></table></div> + +<p>One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.</p> + +<p>An alternative is to define a <span class="emph">custom C type</span> that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don&rsquo;t need to think about applying helper functions and +dereferencing pointers.</p> + +<h1><a name="(part._.Custom_.C_types)"></a>Custom C types</h1> + +<p>I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.</p> + +<p>In this way, it&rsquo;s possible to conduct interesting conversions, such as +dereferencing union objects automatically.</p> + +<p>Now let&rsquo;s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.</p> + +<p>First, we&rsquo;ll start by defining a struct type for the Racket representation +of Cairo paths:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define-struct.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._struct%29%29">struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ptr</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:property</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._prop~3asequence%29%29">prop:sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The representation will store one field <span class="RktWrap"><span class="RktSym">ptr</span></span> which, as the name +suggests, will store a pointer value. We&rsquo;ll see what to do with this +pointer later.</p> + +<p>This definition uses a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/structprops.html#%28tech._structure._type._property%29"><span class="techinside">structure type property</span></a> to make instances of +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> automatically work as sequences. This means that you +can iterate over them with a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span></span> loop or apply <span class="RktWrap"><span class="RktSym">sequence-ref</span></span> +on them. The property takes a function that takes an instance of the struct +type itself (here <span class="RktWrap"><span class="RktSym">p</span></span>) and that returns a sequence.</p> + +<p>We&rsquo;ll later define the <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span> function that will actually +construct the relevant sequence for us. For now, let&rsquo;s see how to construct +the C type given this struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract pointer out of representation</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">racket-&gt;c</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Just apply the Racket constructor</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">c-&gt;racket</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">racket-&gt;c</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">c-&gt;racket</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The base type for this <span class="RktWrap"><span class="RktSym">_cairo_path_t</span></span> is a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type. Since +the Cairo API returns pointers to new path values, it&rsquo;s hard to avoid using some +kind of pointer type as the base type here.</p> + +<p>This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we&rsquo;ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.</p> + +<p>The real work is done by the helper function that makes a +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> instance work as a sequence.</p> + +<p>Starting top-down, let&rsquo;s look at the definition of <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Cairo-Path -&gt; Sequence</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match-define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/values.html#%28def._%28%28quote._~23~25kernel%29._values%29%29">values</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The first thing the function does is extract the pointer out of the +representation, and then immediately calls <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.</p> + +<p><span style="font-weight: bold">Note:</span> in case you&rsquo;re not very familiar with Racket pattern matching, the +<span class="RktWrap"><span class="RktSym">match-define</span></span> form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml&rsquo;s <span class="stt">let</span> statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +<a href="http://docs.racket-lang.org/reference/match.html#%28form._%28%28lib._racket%2Fmatch..rkt%29._match-define%29%29">docs</a>.</p> + +<p>After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> function. Essentially, <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.</p> + +<p><span style="font-weight: bold">Note:</span> technically <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it&rsquo;s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.</p> + +<p>In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the <span class="RktWrap"><span class="RktSym">pos-&gt;element</span></span> function +and its helpers:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer -&gt; Integer -&gt; Element</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract the data path header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Length includes header, so subtract 1</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._sub1%29%29">sub1</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._add1%29%29">add1</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer Integer Integer -&gt; (Listof Data)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">offset argument</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in <span class="RktWrap"><span class="RktSym">get-points</span></span> +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.</p> + +<p>Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Note that determining the end of the sequence is very easy. It&rsquo;s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="stt"> </span><span class="RktSym">pos</span><span class="stt"> </span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></span>.</p> + +<p>Now we can try using a path as a sequence:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Using path as a sequence</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">elem</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="hspace">&nbsp;</span><span class="RktSym">elem</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 50.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (206.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (115.0 115.0))</span></p></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span></span> type.</p> + +<p>That concludes part 3 of the FFI tutorial. Hopefully you&rsquo;re now equipped +to deal with union types and custom C types. If not, see the +<a href="http://docs.racket-lang.org/foreign/index.html">FFI reference</a> +for more details on +<a href="http://docs.racket-lang.org/foreign/C_Union_Types.html">unions</a> +and +<a href="http://docs.racket-lang.org/foreign/ctype.html">custom C types</a>.</p> + +<p><span class="emph">Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!</span></p> + + Tutorial: Racket FFI, Part 2 + + urn:http-prl-ccs-neu-edu:-blog-2016-06-29-tutorial-racket-ffi-part-2 + 2016-06-29T18:48:17Z + 2016-06-29T18:48:17Z + + Asumu Takikawa + +<p>This is part 2 of my tutorial on using the Racket FFI. If you haven&rsquo;t read +part 1 yet, you can find it +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a>. +<span style="font-weight: bold">Update:</span> part 3 is also now available +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.</p> +<!--more--> + +<p>First, here&rsquo;s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">bitmap magic</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_create</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Bitmap -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a helper for displaying the bitmap</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<h1><a name="(part._.Dashes_and_array_arguments)"></a>Dashes and array arguments</h1> + +<p>To start off, let&rsquo;s look at another C example from the Cairo +<a href="https://www.cairographics.org/samples/">samples page</a>. +This time we will look at the "dash" example, which has a +use of an input array:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">sizeof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">)</span><span class="RktMeta">/sizeof</span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktVal">0</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">50.0</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">25.6</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_rel_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">102.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_curve_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The most interesting function here is <span class="stt">cairo_set_dash</span>, which takes an +array argument. The only other new functions are <span class="stt">cairo_rel_line_to</span> +and <span class="stt">cairo_curve_to</span> which have very straightforward C types:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-rel-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_rel_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-curve-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_curve_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Meanwhile, the C type signature for <span class="stt">cairo_set_dash</span> from the Cairo +docs looks like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Something to note about the arguments is that <span class="stt">num_dashes</span> +encodes the length of the array <span class="stt">dashes</span>. This will come up later when +we want to make the C type for this function more convenient.</p> + +<p>On the Racket side, it&rsquo;s natural to represent the array of dashes as either a +list or <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/vectors.html#%28tech._vector%29"><span class="techinside">vector</span></a> of numbers. Given that, a fairly literal translation of +the C type above might look like the following:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This type includes a type constructor we haven&rsquo;t seen yet: <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span>. +This is a so-called <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28tech._custom._function._type%29"><span class="techinside">custom function type</span></a> that has special meaning +inside of a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.</p> + +<p>Since we only want to provide a list from the Racket side to C, we&rsquo;ll use +the <span class="RktWrap"><span class="RktSym">i</span></span> input mode. We can then call the function like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">4</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Note that because of how we defined the type of <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it&rsquo;s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.</p> + +<p>Luckily, the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type constructor actually lets you do this with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">name</span><span class="stt"> </span><span class="RktSym">:</span><span class="stt"> </span><span class="RktSym">type</span><span class="RktPn">)</span></span> syntax for naming arguments in combination with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">type</span><span class="stt"> </span><span class="RktVar">=</span><span class="stt"> </span><span class="RktSym">expr</span><span class="RktPn">)</span></span> syntax for supplying computed arguments:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">name this argument for later uses in the type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a computed argument position</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3d%29%29">=</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>When a computed argument is specified with a <span class="RktWrap"><span class="RktVar">=</span></span>, it&rsquo;s not necessary to provide +the argument on the Racket side. So <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> is now an arity 3 +function that can be called like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This means we&rsquo;ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it&rsquo;s also possible to use Racket vectors instead of lists by using +the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__vector%29%29">_vector</a></span></span> type constructor.</p> + +<p>Putting it all together, we can reproduce the dashes example like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">25.6</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-rel-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-1</span>02.4</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-curve-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Result_arguments_and_.C_structs)"></a>Result arguments and C structs</h1> + +<p>For some more advanced FFI hacking, let&rsquo;s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Our challenge is to make a Racket function that takes a string (let&rsquo;s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let&rsquo;s just scale the +text to fit the width and assume the height will be okay.</p> + +<p>To implement the key step of measuring the text size, we can use the +<a href="https://www.cairographics.org/manual/cairo-text.html#cairo-text-extents"><span class="stt">cairo_text_extents</span></a> +function. Its type signature is as follows:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">char</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*utf8,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*extents</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The interesting part of this signature is that +<a href="https://www.cairographics.org/manual/cairo-cairo-scaled-font-t.html#cairo-text-extents-t"><span class="stt">cairo_text_extents_t</span></a> +is a struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">Cairo</span><span class="hspace">&nbsp;</span><span class="RktCmt">docs</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">height</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>We haven&rsquo;t yet seen how to handle C structs with the FFI, but it&rsquo;s not +too tricky. Support for C structs comes built-in and will look familiar +if you&rsquo;re used to Racket structs. We can directly translate the documented +definition above into a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span> declaration:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the leading underscore is mandatory</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">width</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">height</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">x-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">pointer to struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">allows NULL pointer</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer/null</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr></tbody></table></div> + +<p>Along with functions that look like regular Racket struct operations:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a struct constructor</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:make-cairo_text_extents_t&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field selector</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:cairo_text_extents_t-width&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a predicate for the struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t?</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:^TYPE?&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field mutation function</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">set-cairo_text_extents_t-width!</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:set-cairo_text_extents_t-width!&gt;</span></p></td></tr></tbody></table></div> + +<p>With the struct type defined, it&rsquo;s easy to come up with a rudimentary +interface for <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>This style of programming feels awfully imperative though. Since we&rsquo;re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span> FFI wrapper +by combining named arguments, a new <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> type constructor, +and a neat feature of <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> that lets you customize +the return result:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">named args and _ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">ext</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the return result of the C function</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">custom return result for the wrapper</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">ext</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> constructor works like the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span> constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the <span class="RktWrap"><span class="RktSym">o</span></span> mode to <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span>. +In output mode, this type will automatically allocate a new instance of the type +(using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._malloc%29%29">malloc</a></span></span> function) and arrange for it to be passed in as +a pointer.</p> + +<p>The strangest part of this example is that there are now two uses of the +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span></span> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.</p> + +<p>Using this new version of the wrapper is much simpler:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>With that in hand, it&rsquo;s pretty easy to write the function we set out to write:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-show-text</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_show_text</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-scale</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">String -&gt; Void</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">draws a string scaled horizontally</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">20</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">2.0</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._-%29%29"><span class="nobreak">-</span></a></span><span class="hspace">&nbsp;</span><span class="RktVal">256.0</span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym">width</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-scale</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-show-text</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>And to conclude part 2 of this tutorial, here&rsquo;s an example use +of the new <span class="RktWrap"><span class="RktSym">fit-text</span></span> function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktVal">"Saluton, Mondo / Hallo, mundo"</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + + Tutorial: Using Racket's FFI + + urn:http-prl-ccs-neu-edu:-blog-2016-06-27-tutorial-using-racket-s-ffi + 2016-06-27T16:22:11Z + 2016-06-27T16:22:11Z + + Asumu Takikawa + +<p><span style="font-weight: bold">Update:</span> this post is now part of a series. Part 2 is +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a> +and part 3 is +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>I&rsquo;ve seen several people ask for a tutorial on Racket&rsquo;s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +<a href="http://docs.racket-lang.org/foreign/index.html">documentation</a> +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.</p> + +<p>With that in mind, this blog post will provide a step-by-step tutorial +for Racket&rsquo;s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.</p> +<!--more--> + +<p>Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop <a href="http://www.ccs.neu.edu/racket/pubs/scheme04-bo.pdf">paper</a> +that you can read if you&rsquo;re curious about the design.</p> + +<p>The tutorial will focus on using the <a href="https://www.cairographics.org/">Cairo</a> +graphics library, mainly because it comes bundled with Racket.</p> + +<p>To start, let&rsquo;s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo&rsquo;s +<a href="https://www.cairographics.org/samples/">samples page</a>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">30.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">CAIRO_LINE_CAP_ROUND</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This uses the Racket drawing library <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/draw/index.html"><span class="RktSym">racket/draw</span></a></span> to construct +a bitmap object that we&rsquo;ll draw on. The <span class="RktWrap"><span class="RktSym">get-handle</span></span> method just extracts a +low-level Cairo surface value that we can use.</p> + +<p>NB: these code snippets don&rsquo;t come with a <span class="stt">#lang</span> declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy &amp; paste the snippets into your REPL.</p> + +<p>Our first real step is to import the FFI itself:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>As the module name suggests, the FFI is <span class="emph">unsafe</span> and can cause your Racket process +to segfault. If you&rsquo;re following along in DrRacket, you will want to save your file +frequently.</p> + +<p>Next, we can load the Cairo library to obtain a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28tech._foreign._library._value%29"><span class="techinside">foreign-library value</span></a>, which +is a handle that we use to access C values and functions:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Since Cairo has already been loaded by the Racket process because of the +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/gui/index.html"><span class="RktSym">racket/gui</span></a></span> import earlier, we can supply <span class="RktWrap"><span class="RktVal">#f</span></span> here as an +argument to <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span>. Normally you supply the name of a shared +library file such as <span class="RktWrap"><span class="RktVal">"libcairo"</span></span>:</p> + +<div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"libcairo"</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"2"</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></p></div> + +<p>The last list argument specifies the accepted versions (<span class="RktWrap"><span class="RktVal">#f</span></span> allows +a version-less library). For this post, those details aren&rsquo;t important but +see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span> if you&rsquo;re curious.</p> + +<h1><a name="(part._.Extracting_functions)"></a>Extracting functions</h1> + +<p>Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> function. The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> +function takes three arguments:</p> + +<ul> + <li> + <p>The name of the value as a string (or symbol or bytestring)</p></li> + <li> + <p>a foreign library value, and</p></li> + <li> + <p>a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/types.html#%28tech._c._type%29"><span class="techinside">C type</span></a>, which is a type description that tells +the FFI how to marshall between Racket and C.</p></li></ul> + +<p>C types are a crucial concept for the FFI. They range from relatively +simple types like <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> to more complicated +type constructors such as <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span>. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span></span> with two functions that +handle marshalling between C and Racket code.</p> + +<p>To make progress with our Cairo code, we need to create a drawing context from +the surface object <span class="RktWrap"><span class="RktSym">bt-surface</span></span> that we defined a while ago. The +relevant function in the Cairo docs is +<a href="https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-create"><span class="stt">cairo_create</span></a>, +which has the following type signature:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">NB:</span><span class="hspace">&nbsp;</span><span class="RktCmt">this</span><span class="hspace">&nbsp;</span><span class="RktCmt">is</span><span class="hspace">&nbsp;</span><span class="RktCmt">C</span><span class="hspace">&nbsp;</span><span class="RktCmt">code</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_create</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_surface_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*target</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a <span class="stt">cairo_surface_t</span> and +returns a pointer to a <span class="stt">cairo_t</span>. Let&rsquo;s start with a very simple C type +that matches up with this behavior: </div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span></p></div></div> + +<div class="SIntrapara">This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type.</div> + +<p>The following definition shows how to use this type to obtain a foreign +function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Then we can use <span class="RktWrap"><span class="RktSym">cairo-create</span></span> as an ordinary racket function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">ctx</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Interlude__more_type_safety)"></a>Interlude: more type safety</h1> + +<p>Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> types, it is easy +to accidentally misuse the function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">You may not want to actually run this</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a cairo_t is not a cairo_surface_t</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>To prevent such bad uses, it is good practice to use <span class="emph">tagged</span> pointer types +using <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>. Here are two example definitions that +correspond to the <span class="stt">cairo_t</span> and <span class="stt">cairo_surface_t</span> types from earlier:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">The leading underscores are mandatory</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>We can then redefine <span class="RktWrap"><span class="RktSym">cairo-create</span></span> with a better type, which will +prevent ill-typed calls:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">cairo_surface_t-&gt;C: argument is not non-null</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">`cairo_surface_t' pointer</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktErr">argument: #&lt;cpointer:cairo_t&gt;</span></p></td></tr></tbody></table></div> + +<p>Unfortunately our old definition of <span class="RktWrap"><span class="RktSym">ctx</span></span> doesn&rsquo;t have this tag:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#f</span></p></td></tr></tbody></table></div> + +<p>Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it&rsquo;s also possible to update +existing pointers with a tag like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-push-tag%21%29%29">cpointer-push-tag!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#t</span></p></td></tr></tbody></table></div> + +<p>Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).</p> + +<h1><a name="(part._.Macros_for_reducing_boilerplate)"></a>Macros for reducing boilerplate</h1> + +<p>Now let&rsquo;s start building the FFI bindings for the functions in the Cairo sample. +First, let&rsquo;s go ahead and look at all of the types for the sample functions +from the C API docs:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_cap_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">line_cap</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Starting with <span class="stt">cairo_move_to</span>, we can set up a definition like we did +with <span class="stt">cairo_create</span> before:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">"cairo_move_to"</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span> library that help reduce the +verbosity. Here&rsquo;s an alternative definition of <span class="RktWrap"><span class="RktSym">cairo-move-to</span></span> +using the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form from +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>As you can see, the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +<span class="RktWrap"><span class="RktSym">cairo_move_to</span></span>) you also don&rsquo;t need to supply the C name either.</p> + +<p>The definitions for <span class="stt">cairo_line_to</span>, <span class="stt">cairo_set_line_width</span>, and +<span class="stt">cairo_stroke</span> aren&rsquo;t very interesting, so I&rsquo;ll just include them +below without comment:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="stt">cairo_set_line_cap</span> case is more interesting because the type +<span class="stt">cairo_line_cap_t</span> is a C enumeration type. Racket&rsquo;s FFI comes with +convenience forms for defining enumeration types&#8212; + <wbr />though it&rsquo;s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here&rsquo;s a quote from the Barzilay and Orlovsky +paper on that:</p> + +<blockquote class="SubFlow"> + <p>Our design follows a simple principle: keep C-level +functionality to a minimum.</p></blockquote> + +<p>and specifically about enumerations:</p> + +<blockquote class="SubFlow"> + <p>For example, the C level part of our interface does not commit to a +specific implementation for enumerations &#8212; it simply exposes C integers.</p></blockquote> + +<p>To define an enumeration, we can use the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the <span class="stt">cairo_line_cap_t</span> type, it suffices to +just supply the cases as a list of symbols:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It&rsquo;s also possible to specify how the symbols map to integers more precisely +(see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> for those details).</p> + +<p>Given this type, we can specify the type for the line cap function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<h1><a name="(part._.Putting_it_all_together)"></a>Putting it all together</h1> + +<p>Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">30.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-cap</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">round</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object <span class="RktWrap"><span class="RktSym">bt</span></span>. To see the +results of what we drew, we can just evaluate <span class="RktWrap"><span class="RktSym">bt</span></span> at the REPL. But it&rsquo;s +a little nicer if we use the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> library to draw a frame around +it to distinguish it from the background:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>And we&rsquo;re done! Of course, there is a lot more to the FFI. For example, I haven&rsquo;t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I&rsquo;m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!</p> \ No newline at end of file diff --git a/blog/feeds/Racket.rss.xml b/blog/feeds/Racket.rss.xml new file mode 100644 index 00000000..d26e1201 --- /dev/null +++ b/blog/feeds/Racket.rss.xml @@ -0,0 +1,3314 @@ + + + + PRL Blog: Posts tagged 'Racket' + PRL Blog: Posts tagged 'Racket' + http://prl.ccs.neu.edu/blog/tags/Racket.html + Fri, 27 Apr 2018 21:35:22 UT + Fri, 27 Apr 2018 21:35:22 UT + 1800 + + The Racket School 2018: Create your own language + http://prl.ccs.neu.edu/blog/2018/04/27/the-racket-school-2018-create-your-own-language/?utm_source=Racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-04-27-the-racket-school-2018-create-your-own-language + Fri, 27 Apr 2018 21:35:22 UT + Ben Greenman + +<p>The Racket School 2018: Create your own language • 9–13 July • Salt Lake City</p> +<!-- more--> + +<p>The Racket team has spent over thirty years developing and refining a coherent intellectual tradition for studying and building programming languages. This year’s school will introduce participants to Racket’s framework for language-oriented programming, which the summer school faculty recently spelled out in a a cover article in the <a href="https://tinyurl.com/RacketCACM">Communications of the ACM</a></p> + +<p>Concretely, the 2018 Racket Summer School will cover the following topics:</p> + +<ul> + <li>the spectrum of programming languages;</li> + <li>modules and syntax, or languages as libraries;</li> + <li>DrRacket’s support for language-oriented programming;</li> + <li>a domain-specific language for adding types to languages;</li> + <li>tools and techniques for implementing notational conveniences; and</li> + <li>research challenges in language-oriented programming.</li></ul> + +<p>If these topics intrigue you, attend the Racket Summer School:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2018/">http://summer-school.racket-lang.org/2018/</a></li></ul> + +<p>This is not your run-of-the-mill summer school. We will do our best to make it exciting, entertaining, and useful to a broad spectrum of attendees, both academic and industrial.</p> + +<p>P.S. We will send you your first problem set in June, a month before the summer school to whet your appetite.</p> + + Tutorial: Zero to Sixty in Racket + http://prl.ccs.neu.edu/blog/2016/08/02/tutorial-zero-to-sixty-in-racket/?utm_source=Racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-08-02-tutorial-zero-to-sixty-in-racket + Tue, 02 Aug 2016 01:29:11 UT + Ben Greenman + +<p>Racket is excellent for incrementally growing scripts into full-fledged programs. +This post steps through the evolution of one small program and highlights the + Racket tools that enable incremental advances.</p> +<!--more--> + +<p></p> + +<div class="SIntrapara">Why should anyone use <a href="http://racket-lang.org/">Racket</a>? +There are two reasons: +</div> + +<div class="SIntrapara"> + <ol> + <li> + <p>You have a problem that can only be solved with Racket&rsquo;s language-building tools</p></li> + <li> + <p>Racket is a nice language to program in. +(Has lexical scope, parentheses, <a href="http://con.racket-lang.org">active users</a>...)</p></li></ol></div> + +<p>My favorite part of Racket is how it supports a certain development style of + evolving scripts into programs. +When I start coding (after design, before debugging), I can focus the problem at hand. +Next come examples, unit tests, and types to be sure the solution is correct. +Finally, I worry about aesthetics, efficiency, and how the solution can be used as a library in a larger context.</p> + +<p>Bottom line: with Racket, my coding is aligned with my priorities. +And as I transition from "no code" to "working code" to "robust code" to "re-usable code", + the program is almost always runnable.</p> + +<h1><a name="(part._.Problem__.A_.K.W.I.C_.Index_.Production_.System)"></a>Problem: A KWIC Index Production System</h1> + +<p>A KWIC index system +reads input from a file, +divides each line of the file into whitespace-separated words, +and outputs (in alphabetical order) all circular shifts of all lines.</p> + +<p>The first circular shift of a line <span class="RktWrap"><span class="RktVal">"A B C"</span></span> is the line <span class="RktWrap"><span class="RktVal">"B C A"</span></span>. +The second circular shift is <span class="RktWrap"><span class="RktVal">"C A B"</span></span>.</p> + +<p>Building a KWIC index is a historical problem. +According to <a href="https://www.cs.umd.edu/class/spring2003/cmsc838p/Design/criteria.pdf">D.L. Parnas (1972)</a>:</p> + +<blockquote class="SubFlow"> + <p>Except under extreme circumstances (huge data base, no supporting software) +such a system could be implemented by a good programmer within a week or two.</p></blockquote> + +<p>See also: <a href="https://yanniss.github.io/law.html">Yannis&rsquo;s Law</a>.</p> + +<p>Today, I bet only <a href="http://wiki.portal.chalmers.se/agda/pmwiki.php">Agda</a> and <a href="https://scratch.mit.edu/">Scratch</a> + programmers would need the full two weeks. +We&rsquo;ll be done in 20 minutes.</p> + +<h1><a name="(part._.A_.Script)"></a>A Script</h1> + +<p>To start, open a file and type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>You can name the file anything, like <span class="stt">kwic.rkt</span> or <span class="stt">rkt.kwic</span> or <span class="stt">foo</span>. +Racket doesn&rsquo;t care, + but it does need the <span class="stt">#lang</span> line to read the contents of the file.</p> + +<p>Though, you should use the <span class="stt">.rkt</span> extension.</p> + +<p>The first part of the solution is a function to read input from a file into a + list of strings for further processing. +The built-in function <a href="http://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28lib._racket%2Ffile..rkt%29._file-~3elines%29%29">file-&gt;lines</a> + does exactly this, but we&rsquo;ll for-loop instead.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="hspace">&nbsp;</span><span class="RktSym">filename</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="hspace">&nbsp;</span><span class="RktSym">filename</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>When called with a filename like <span class="RktWrap"><span class="RktVal">"heart-of-darkness.txt"</span></span>, the function + uses <a href="http://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a> to build a list of lines by reading from a port + with <a href="http://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a>. +The port is the data from <span class="RktWrap"><span class="RktSym">filename</span></span>, thanks to <a href="http://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a>.</p> + +<p>Next is a function to convert a list of strings into a list of lists of words. +Here we&rsquo;ll just use library functions.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktSym">lines</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="hspace">&nbsp;</span><span class="RktSym">string-split</span><span class="hspace">&nbsp;</span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>By default, <a href="http://docs.racket-lang.org/reference/strings.html#%28def._%28%28lib._racket%2Fstring..rkt%29._string-split%29%29">string-split</a> divides a string into a list of whitespace-separated substrings. +You can always supply a different delimiter, or use <a href="http://docs.racket-lang.org/reference/regexp.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._regexp-split%29%29">regexp-split</a> + to divide by a regular expression.</p> + +<p>Two tasks left! +First we generate all circular shifts for a list of strings <span class="RktWrap"><span class="RktSym">words</span></span> + by folding up a list with one shift of <span class="RktWrap"><span class="RktSym">words</span></span> for each word.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Second, we alphabetize and print the shifts.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">destruct multiple values</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">first list empty, don</span><span class="RktCmt">'</span><span class="RktCmt">t care about second</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#t</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">first list non-empty, second empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1-rest</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="hspace">&nbsp;</span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktVal">" "</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">for-each is like map, but returns (void)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="hspace">&nbsp;</span><span class="RktSym">display-words</span><span class="hspace">&nbsp;</span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Gluing it all together, here&rsquo;s the full script (with type annotations in comments).</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">type</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">=</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">type</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">=</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Path-String</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Move</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">element</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">last</span><span class="hspace">&nbsp;</span><span class="RktCmt">position</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lexicographic</span><span class="hspace">&nbsp;</span><span class="RktCmt">order</span><span class="hspace">&nbsp;</span><span class="RktCmt">on</span><span class="hspace">&nbsp;</span><span class="RktCmt">equal-length</span><span class="hspace">&nbsp;</span><span class="RktCmt">lists</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Boolean</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">don't</span><span class="hspace">&nbsp;</span><span class="RktCmt">care</span><span class="hspace">&nbsp;</span><span class="RktCmt">about</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">non-empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/booleans.html#%28def._%28%28quote._~23~25kernel%29._not%29%29">not</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._null~3f%29%29">null?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"</span><span class="hspace">&nbsp;</span><span class="RktVal">"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Path-String</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">append*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">End-to-end</span><span class="hspace">&nbsp;</span><span class="RktCmt">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">run-test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"test.txt"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Make</span><span class="hspace">&nbsp;</span><span class="RktCmt">a</span><span class="hspace">&nbsp;</span><span class="RktCmt">file</span><span class="hspace">&nbsp;</span><span class="RktCmt">and</span><span class="hspace">&nbsp;</span><span class="RktCmt">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/when_unless.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._unless%29%29">unless</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._file-exists~3f%29%29">file-exists?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">run-test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Running the file should print:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<h1><a name="(part._.Testing_and_.Submodules)"></a>Testing and Submodules</h1> + +<p>Any top-level expressions in a file can work as unit tests. +The <a href="http://docs.racket-lang.org/reference/booleans.html%3F#%28def._%28%28quote._~23~25kernel%29._equal~3f%29%29">equal?</a> statement below checks whether the first circular shift + of <span class="RktWrap"><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="stt"> </span><span class="RktVal">"B"</span><span class="stt"> </span><span class="RktVal">"C"</span><span class="RktVal">)</span></span> is <span class="RktWrap"><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="stt"> </span><span class="RktVal">"C"</span><span class="stt"> </span><span class="RktVal">"A"</span><span class="RktVal">)</span></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Equality.html#%28def._%28%28quote._~23~25kernel%29._equal~3f%29%29">equal?</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Running the file now prints <span class="RktWrap"><span class="RktVal">#t</span></span> to the console, meaning the test passed. +We can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/exns.html#%28def._%28%28quote._~23~25kernel%29._error%29%29">error</a></span></span> or <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/exns.html#%28def._%28%28quote._~23~25kernel%29._raise-user-error%29%29">raise-user-error</a></span></span> to make failures easier + to notice. +Or we can use the <a href="http://docs.racket-lang.org/rackunit/api.html">RackUnit</a> testing library.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">import the testing library</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>These tests run each time the module does. +If you prefer to run tests only in a specific context, and not when the + module is run or imported as a library, you can move them to a separate + file or into a <a href="http://docs.racket-lang.org/reference/eval-model.html#%28tech._submodule%29">submodule</a>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Open a submodule named </span><span class="RktCmt">'</span><span class="RktCmt">test</span><span class="RktCmt">'</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Running the module normally via <span class="stt">racket kwic.rkt</span> will not run code + in the submodule. +Instead, use <span class="stt">raco test</span> to run the tests.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">raco</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">raco</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._submod%29%29">submod</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"kwic.rkt"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">passed</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The reason we used <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span></span>, instead of Racket&rsquo;s <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%29%29">module</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%2A%29%29">module*</a></span></span> + forms is that <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span></span> inherits the language and namespace of its + containing module and can be incrementally extended. +This way, we can keep tests near the relevant code.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"is"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"as"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"does"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"as"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"does"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"is"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p><a href="http://docs.racket-lang.org/rackunit/api.html">RackUnit</a> in a + separate file or <span class="RktWrap"><span class="RktSym">test</span></span> submodule is the unofficial standard for testing + Racket programs.</p> + +<h1><a name="(part._.Recognizing_.Patterns__.Avoiding_.Repetition)"></a>Recognizing Patterns, Avoiding Repetition</h1> + +<p>Every unit test we&rsquo;ve written uses <a href="http://docs.racket-lang.org/rackunit/api.html#%28def._%28%28lib._rackunit%2Fmain..rkt%29._check-equal~3f%29%29">check-equal?</a>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"hello"</span><span class="hspace">&nbsp;</span><span class="RktVal">"world"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">" lost "</span><span class="hspace">&nbsp;</span><span class="RktVal">" in "</span><span class="hspace">&nbsp;</span><span class="RktVal">"space"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"lost"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"in"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"space"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"something"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>These tests follow a simple pattern that we can express as a <span class="emph">syntax rule</span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._define-syntax-rule%29%29">define-syntax-rule</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="hspace">&nbsp;</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="RktPn">]</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/begin.html#%28form._%28%28quote._~23~25kernel%29._begin%29%29">begin</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"hello"</span><span class="hspace">&nbsp;</span><span class="RktVal">"world"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">" out "</span><span class="hspace">&nbsp;</span><span class="RktVal">" in "</span><span class="hspace">&nbsp;</span><span class="RktVal">"the ozone"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"out"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"in"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"the"</span><span class="hspace">&nbsp;</span><span class="RktVal">"ozone"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"something"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span></span> are not pseudocode! +They denote Kleene-star repetition, like a sextile (<span class="RktWrap"><span class="RktVal">"*"</span></span>) in a regular expression. +In this case, the input pattern is a sequence of lists with two S-expressions, <span class="RktWrap"><span class="RktSym">i</span></span> and + <span class="RktWrap"><span class="RktSym">o</span></span>. +Uses of <span class="RktWrap"><span class="RktSym">i</span></span> and <span class="RktWrap"><span class="RktSym">o</span></span> in the rule must be followed by one <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span></span> to splice + the captured S-expressions into the result.</p> + +<p>Many languages offer higher-order functions and polymorphism to abstract common + behaviors. +Syntax extensions are a different way to avoid repeating yourself. +After 30 years, we are still discovering what syntax extensions are useful for.</p> + +<p>See this <a href="https://groups.google.com/forum/#!topic/racket-users/ss20lwfUhjs/discussion">recent Racket mailing list post</a> for some applications.</p> + +<h1><a name="(part._.Adding_.Static_.Types)"></a>Adding Static Types</h1> + +<p>Changing the <a href="http://docs.racket-lang.org/reference/reader.html#%28idx._%28gentag._79._%28lib._scribblings%2Freference%2Freference..scrbl%29%29%29"><span class="stt">#lang</span></a> line to <span class="RktWrap"><span class="RktSym">typed/racket</span></span> adds static type-checking to our program. +If we only change the language and run the code as-is, there will be type errors. +But we can use submodules again to incrementally check our design with types.</p> + +<p>Note: <a href="http://docs.racket-lang.org/ts-reference/Typed_Regions.html">typed regions</a> + are another way to embed typed code into untyped contexts.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%29%29">module</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">typed/racket</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Need</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">annotate:</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-</span><span class="hspace">&nbsp;</span><span class="RktCmt">function</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameters</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-</span><span class="hspace">&nbsp;</span><span class="RktCmt">for-loop</span><span class="hspace">&nbsp;</span><span class="RktCmt">return</span><span class="hspace">&nbsp;</span><span class="RktCmt">types</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Next</span><span class="hspace">&nbsp;</span><span class="RktCmt">migration</span><span class="hspace">&nbsp;</span><span class="RktCmt">step:</span><span class="hspace">&nbsp;</span><span class="RktCmt">move</span><span class="hspace">&nbsp;</span><span class="RktCmt">other</span><span class="hspace">&nbsp;</span><span class="RktCmt">untyped</span><span class="hspace">&nbsp;</span><span class="RktCmt">functions</span><span class="hspace">&nbsp;</span><span class="RktCmt">here</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._all-defined-out%29%29">all-defined-out</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">t</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">&lt;rest</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">file</span><span class="hspace">&nbsp;</span><span class="RktCmt">omitted&gt;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>After scooping all functions into the Typed Racket bubble, we can remove the + submodule declaration and change <span class="stt">#lang racket</span> to <span class="stt">#lang typed/racket</span>.</p> + +<h1><a name="(part._.Finally__a_.Library)"></a>Finally, a Library</h1> + +<p>Other modules can import our functions if we use a <a href="http://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a> statement. +By <a href="https://docs.racket-lang.org/style/Units_of_Code.html">convention</a>, exports belong at the top of a file.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">#lang</span><span class="hspace">&nbsp;</span><span class="RktMeta">typed/racket</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-index</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">&lt;definitions</span><span class="hspace">&nbsp;</span><span class="RktCmt">here&gt;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Then any typed or untyped module can use <span class="RktWrap"><span class="RktSym">kwic-index</span></span> by writing + <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="stt"> </span><span class="RktVal">"kwic.rkt"</span><span class="RktPn">)</span></span>.</p> + +<p>As a finishing touch, we can use the <a href="http://docs.racket-lang.org/reference/Command-Line_Parsing.html">racket/cmdline</a> library + inside a <span class="stt">main</span> submodule to give a basic front-end interface. +Similar to <span class="stt">module+ test</span>, a <span class="stt">module+ main</span> declares code that + inherits the file&rsquo;s bindings and language but is only run when the program + is executaed.</p> + +<p>Here is the complete typed and tested code listing. +The <span class="RktWrap"><span class="RktSym">main</span></span> submodule is at the bottom.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/ts-reference/index.html"><span class="RktSym">typed/racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">typed/rackunit</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._define-syntax-rule%29%29">define-syntax-rule</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">o</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/begin.html#%28form._%28%28quote._~23~25kernel%29._begin%29%29">begin</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">o</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">define-type</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">define-type</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">make-temporary-file</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"The</span><span class="hspace">&nbsp;</span><span class="RktVal">Nellie,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"a</span><span class="hspace">&nbsp;</span><span class="RktVal">cruising</span><span class="hspace">&nbsp;</span><span class="RktVal">yawl,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"swung</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">her</span><span class="hspace">&nbsp;</span><span class="RktVal">anchor</span><span class="hspace">&nbsp;</span><span class="RktVal">without</span><span class="hspace">&nbsp;</span><span class="RktVal">a</span><span class="hspace">&nbsp;</span><span class="RktVal">flutter</span><span class="hspace">&nbsp;</span><span class="RktVal">of</span><span class="hspace">&nbsp;</span><span class="RktVal">sails,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"and</span><span class="hspace">&nbsp;</span><span class="RktVal">was</span><span class="hspace">&nbsp;</span><span class="RktVal">at</span><span class="hspace">&nbsp;</span><span class="RktVal">rest."</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">actual</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expect</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">file-&gt;lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._delete-file%29%29">delete-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">actual</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expect</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#{</span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta">::</span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"hello"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"world"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Move</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">element</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">last</span><span class="hspace">&nbsp;</span><span class="RktCmt">position</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lexicographic</span><span class="hspace">&nbsp;</span><span class="RktCmt">order</span><span class="hspace">&nbsp;</span><span class="RktCmt">on</span><span class="hspace">&nbsp;</span><span class="RktCmt">equal-length</span><span class="hspace">&nbsp;</span><span class="RktCmt">lists</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Boolean</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">don't</span><span class="hspace">&nbsp;</span><span class="RktCmt">care</span><span class="hspace">&nbsp;</span><span class="RktCmt">about</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">non-empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"</span><span class="hspace">&nbsp;</span><span class="RktVal">"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._open-output-string%29%29">open-output-string</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._get-output-string%29%29">get-output-string</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A\nB</span><span class="hspace">&nbsp;</span><span class="RktVal">C\n"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"D"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"D"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">append*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._open-output-string%29%29">open-output-string</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">make-temporary-file</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._delete-file%29%29">delete-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._get-output-string%29%29">get-output-string</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">string-join</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"if</span><span class="hspace">&nbsp;</span><span class="RktVal">this</span><span class="hspace">&nbsp;</span><span class="RktVal">imagine"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"this</span><span class="hspace">&nbsp;</span><span class="RktVal">imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to\n"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"\n"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">main</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket/cmdline</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Parameterof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Any</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28def._%28%28quote._~23~25kernel%29._make-parameter%29%29">make-parameter</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">command-line</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:program</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"kwic</span><span class="hspace">&nbsp;</span><span class="RktVal">index"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:once-each</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktVal">"-o"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"--output"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">user-supplied</span><span class="hspace">&nbsp;</span><span class="RktCmt">input</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"Write</span><span class="hspace">&nbsp;</span><span class="RktVal">output</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">file"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">update</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameter</span><span class="hspace">&nbsp;</span><span class="RktCmt">*output-to*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:args</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">*output-to*</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">read</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameter</span><span class="hspace">&nbsp;</span><span class="RktCmt">*output-to*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3f%29%29">string?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._open-output-file%29%29">open-output-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">cast</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/when_unless.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._when%29%29">when</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3f%29%29">string?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._close-output-port%29%29">close-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">Sample interactions: +</div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">kwic</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expects</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">&lt;file-name&gt;</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">on</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">the</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">command</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quasiquote.html#%28form._%28%28quote._~23~25kernel%29._unquote%29%29">,</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">given</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">arguments</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">echo</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"It</span><span class="hspace">&nbsp;</span><span class="RktVal">is</span><span class="hspace">&nbsp;</span><span class="RktVal">a</span><span class="hspace">&nbsp;</span><span class="RktVal">truth</span><span class="hspace">&nbsp;</span><span class="RktVal">universally</span><span class="hspace">&nbsp;</span><span class="RktVal">acknowledged"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">pride-and-prejudice.txt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-o</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">pride-and-prejudice.txt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">wc</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-l</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktVal">6</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span></td></tr></tbody></table></div></div> + +<h1><a name="(part._.Closing)"></a>Closing</h1> + +<p>We started with functions, wrote (and quarantined) unit tests, + reinforced our design with types, and added a command-line interface. +Going forward we could add <a href="http://docs.racket-lang.org/scribble/index.html">Scribble</a> documentation and share our work as a <a href="http://pkgn.racket-lang.org/">package</a>.</p> + +<p></p> + +<div class="SIntrapara">For more on building languages with Racket: +</div> + +<div class="SIntrapara"> + <ul> + <li> + <p><a href="http://www.hashcollision.org/brainfudge/">Fudging up a Racket (html)</a></p></li> + <li> + <p><a href="http://dl.acm.org/authorize?6529547">Creating Languages in Racket (pdf)</a></p></li> + <li> + <p><a href="http://www.ccs.neu.edu/home/matthias/manifesto/">The Racket Manifesto (html)</a></p></li> + <li> + <p><a href="http://www.terohasu.net/hasu-flatt--els16--preprint.pdf">Source-to-Source Compilation via Submodules (pdf)</a></p></li></ul></div> + + Tutorial: Racket FFI, part 3 + http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/?utm_source=Racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-07-11-tutorial-racket-ffi-part-3 + Mon, 11 Jul 2016 17:33:40 UT + Asumu Takikawa + +<p>This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a> +and part 2 +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a>.</p> + +<p>In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.</p> +<!--more--> + +<p>As in the second post, let&rsquo;s start with some prologue code that establishes +the definitions from the previous two posts. But first, I&rsquo;m getting tired of +writing the <span class="stt">#:c-id identifier</span> notation for the underscored C function +names.</p> + +<p>Instead, let&rsquo;s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">$</span><span class="hspace">&nbsp;</span><span class="RktMeta">raco</span><span class="hspace">&nbsp;</span><span class="RktMeta">pkg</span><span class="hspace">&nbsp;</span><span class="RktMeta">install</span><span class="hspace">&nbsp;</span><span class="RktMeta">ffi-definer-convention</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>or you can just execute the following snippet in Racket:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pkg</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">pkg-install-command</span><span class="hspace">&nbsp;</span><span class="RktPn">#:skip-installed</span><span class="hspace">&nbsp;</span><span class="RktVal">#t</span><span class="hspace">&nbsp;</span><span class="RktVal">"ffi-definer-convention"</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Downloading repository git://github.com/takikawa/racket-ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: version: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: platform: x86_64-linux [3m]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: target machine: racket</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: installation name: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: variants: 3m</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main collects: /usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: collects paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main pkgs: /usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: pkgs paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: links files: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main docs: /usr/share/racket/doc</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- updating info-domain tables ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- pre-installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing foreign libraries --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing shared files ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- compiling collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- parallel build using 4 jobs ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 making: &lt;pkgs&gt;/ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- creating launchers --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing man pages --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- building documentation --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 running: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 rendering: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 2 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/local-redirect.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 1 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/release.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/search.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/start.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- post-installing collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This will install the package and compile its contents. If you&rsquo;re curious, the +docs for the package are available +<a href="http://docs.racket-lang.org/ffi-definer-convention/index.html">here</a>.</p> + +<p><span style="font-weight: bold">Note:</span> if you&rsquo;ve never installed a package before, you may want to glance +at the +<a href="http://docs.racket-lang.org/pkg/getting-started.html">package system docs</a>. +A tl;dr of packages is that they bundle Racket <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/collects.html#%28tech._collection%29"><span class="techinside">collections</span></a>, which are +sets of modules that you can refer to in a location-independent fashion such as +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> or <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28mod-path._racket%2Flist%29"><span class="RktSym">racket/list</span></a></span>.</p> + +<p>Anyhow, here&rsquo;s the prologue code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">avoid conflict with below</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._except-in%29%29">except-in</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">define-ffi-definer</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the new 3rd-party pkg</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi-definer-convention</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">describes how to transform from</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Racket to C ids</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:make-c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">convention:hyphen-&gt;underscore</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">note lack of #:c-id keyword arguments</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">(_cairo_t -&gt; Void) -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">do some drawing and give us the pict</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">f</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">f</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Notice that the <span class="RktWrap"><span class="RktSym">define-cairo</span></span> forms don&rsquo;t have any <span class="stt">#:c-id</span> keywords +anymore. Instead, the prologue code uses an overriden <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> +from my package that supports a <span class="stt">#:make-c-id</span> keyword that lets you specify +a naming convention to follow.</p> + +<p>Also, instead of creating a single bitmap and drawing into it, we now have a +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> function that takes a drawing function. When called, +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> will call the given function with a new bitmap object and +return the result.</p> + +<p>Now let&rsquo;s get to the main point of this blog post. Let&rsquo;s say that we want to play +with Cairo <a href="https://www.cairographics.org/manual/cairo-Paths.html">path</a> +objects this time. A path is +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-t">defined</a> +as a struct with the following structure:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_status_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">status</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it&rsquo;s +useful to define C types for the types of values in the path struct&rsquo;s fields. First, +let&rsquo;s specify that a <span class="stt">cairo_status_t</span> is an integer type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>It&rsquo;s actually an enum, but for the examples in this post we don&rsquo;t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-data-t">path data objects</a>. +Each path data object is a <span class="stt">cairo_path_data_t</span>, +which is specified with a C union:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">union</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">_cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_type_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">type</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">length</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">header</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">point</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Helpfully, the FFI library comes with support for unions with the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span></span> type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It&rsquo;s pretty +straightforward to specify this type too:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the path data type is just an enum</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">move-to</span><span class="hspace">&nbsp;</span><span class="RktVal">line-to</span><span class="hspace">&nbsp;</span><span class="RktVal">curve-to</span><span class="hspace">&nbsp;</span><span class="RktVal">close-path</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the header case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the point case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>There&rsquo;s a new type constructor here so let me explain that first. +The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span></span> constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span>, this constructor doesn&rsquo;t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.</p> + +<p>Each of the path data structs in the path data array will be manipulated with +the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">create a union from a list of doubles</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1.3</span><span class="hspace">&nbsp;</span><span class="RktVal">5.8</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">source type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">target type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;union&gt;</span></p></td></tr></tbody></table></div> + +<p>This snippet first construct a union object (via the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> +type) using a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span></span>. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it&rsquo;s an easy way +to generate a union object.</p> + +<p>The second line shows that a union prints as an opaque object. You can&rsquo;t do +anything with a union in Racket unless you project it to one of the sub-cases with +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></span> function. +This projection is <span class="emph">unsafe</span>, in the sense that if you don&rsquo;t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.</p> + +<p>More concretely, let&rsquo;s see what happens if we try to extract a value out of the +union both correctly and incorrectly:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">correct (matches construction)</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">cases are zero-indexed and ordered as written</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(1.3 5.8)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">incorrect, error</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">enum:int-&gt;_cairo_path_data_type_t: expected a known</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">#&lt;ctype:ufixint&gt;, got: 3435973837</span></p></td></tr></tbody></table></div> + +<p>Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.</p> + +<p>With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.</p> + +<p>With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it&rsquo;s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.</p> + +<p>So before moving on, let&rsquo;s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we&rsquo;ll figure out how to +deal with the array itself.</p> + +<h1><a name="(part._.Some_low-level_operations)"></a>Some low-level operations</h1> + +<p>Since we still don&rsquo;t have a C type for <span class="stt">cairo_path_t</span>, let&rsquo;s go ahead +and make a simple one where we punt on the work of specifying the array +type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In this type, we have specified the array as a bare <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span>. +For some added safety, we could also use something like +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__cpointer%29%29">_cpointer</a></span><span class="stt"> </span><span class="RktVal">'</span><span class="RktVal">cairo_status_t</span><span class="RktPn">)</span></span>, which sets up a tagged pointer +type like we saw in the first blog post with +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>.</p> + +<p>We&rsquo;ve seen the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type before, but haven&rsquo;t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.</p> + +<p>Before we get to that, let&rsquo;s go ahead and set up an FFI binding for +<span class="stt">cairo_copy_path</span> so that we can obtain a path struct to manipulate:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Do stuff to make the current</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">path non-empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Get the current path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/set_.html#%28form._%28%28quote._~23~25kernel%29._set%21%29%29">set!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Stroke clears the path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">so do it last</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<p>Note that <span class="RktWrap"><span class="RktSym">cairo-copy-path</span></span> gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span>, which +lets you dereference a pointer and access it at some concrete C type.</p> + +<p><span style="font-weight: bold">Note:</span> the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> function also takes an optional +offset argument which we will be used in an example later.</p> + +<p>For example, we can use <span class="RktWrap"><span class="RktSym">a-path</span></span> as a <span class="RktWrap"><span class="RktSym">_simple_cairo_path_t</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(0 #&lt;cpointer&gt; 8)</span></p></td></tr></tbody></table></div> + +<p>And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> again with an array type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">array</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">length field</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">array</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(#&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt;)</span></p></td></tr></tbody></table></div> + +<p>The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(move-to 2)</span></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(50.0 50.0)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">nonsense data here, wrong union case</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(4.2439915824246e-314 0.0)</span></p></td></tr></tbody></table></div> + +<p>One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.</p> + +<p>An alternative is to define a <span class="emph">custom C type</span> that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don&rsquo;t need to think about applying helper functions and +dereferencing pointers.</p> + +<h1><a name="(part._.Custom_.C_types)"></a>Custom C types</h1> + +<p>I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.</p> + +<p>In this way, it&rsquo;s possible to conduct interesting conversions, such as +dereferencing union objects automatically.</p> + +<p>Now let&rsquo;s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.</p> + +<p>First, we&rsquo;ll start by defining a struct type for the Racket representation +of Cairo paths:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define-struct.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._struct%29%29">struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ptr</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:property</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._prop~3asequence%29%29">prop:sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The representation will store one field <span class="RktWrap"><span class="RktSym">ptr</span></span> which, as the name +suggests, will store a pointer value. We&rsquo;ll see what to do with this +pointer later.</p> + +<p>This definition uses a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/structprops.html#%28tech._structure._type._property%29"><span class="techinside">structure type property</span></a> to make instances of +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> automatically work as sequences. This means that you +can iterate over them with a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span></span> loop or apply <span class="RktWrap"><span class="RktSym">sequence-ref</span></span> +on them. The property takes a function that takes an instance of the struct +type itself (here <span class="RktWrap"><span class="RktSym">p</span></span>) and that returns a sequence.</p> + +<p>We&rsquo;ll later define the <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span> function that will actually +construct the relevant sequence for us. For now, let&rsquo;s see how to construct +the C type given this struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract pointer out of representation</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">racket-&gt;c</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Just apply the Racket constructor</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">c-&gt;racket</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">racket-&gt;c</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">c-&gt;racket</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The base type for this <span class="RktWrap"><span class="RktSym">_cairo_path_t</span></span> is a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type. Since +the Cairo API returns pointers to new path values, it&rsquo;s hard to avoid using some +kind of pointer type as the base type here.</p> + +<p>This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we&rsquo;ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.</p> + +<p>The real work is done by the helper function that makes a +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> instance work as a sequence.</p> + +<p>Starting top-down, let&rsquo;s look at the definition of <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Cairo-Path -&gt; Sequence</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match-define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/values.html#%28def._%28%28quote._~23~25kernel%29._values%29%29">values</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The first thing the function does is extract the pointer out of the +representation, and then immediately calls <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.</p> + +<p><span style="font-weight: bold">Note:</span> in case you&rsquo;re not very familiar with Racket pattern matching, the +<span class="RktWrap"><span class="RktSym">match-define</span></span> form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml&rsquo;s <span class="stt">let</span> statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +<a href="http://docs.racket-lang.org/reference/match.html#%28form._%28%28lib._racket%2Fmatch..rkt%29._match-define%29%29">docs</a>.</p> + +<p>After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> function. Essentially, <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.</p> + +<p><span style="font-weight: bold">Note:</span> technically <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it&rsquo;s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.</p> + +<p>In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the <span class="RktWrap"><span class="RktSym">pos-&gt;element</span></span> function +and its helpers:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer -&gt; Integer -&gt; Element</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract the data path header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Length includes header, so subtract 1</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._sub1%29%29">sub1</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._add1%29%29">add1</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer Integer Integer -&gt; (Listof Data)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">offset argument</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in <span class="RktWrap"><span class="RktSym">get-points</span></span> +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.</p> + +<p>Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Note that determining the end of the sequence is very easy. It&rsquo;s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="stt"> </span><span class="RktSym">pos</span><span class="stt"> </span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></span>.</p> + +<p>Now we can try using a path as a sequence:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Using path as a sequence</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">elem</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="hspace">&nbsp;</span><span class="RktSym">elem</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 50.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (206.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (115.0 115.0))</span></p></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span></span> type.</p> + +<p>That concludes part 3 of the FFI tutorial. Hopefully you&rsquo;re now equipped +to deal with union types and custom C types. If not, see the +<a href="http://docs.racket-lang.org/foreign/index.html">FFI reference</a> +for more details on +<a href="http://docs.racket-lang.org/foreign/C_Union_Types.html">unions</a> +and +<a href="http://docs.racket-lang.org/foreign/ctype.html">custom C types</a>.</p> + +<p><span class="emph">Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!</span></p> + + Tutorial: Racket FFI, Part 2 + http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/?utm_source=Racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-29-tutorial-racket-ffi-part-2 + Wed, 29 Jun 2016 18:48:17 UT + Asumu Takikawa + +<p>This is part 2 of my tutorial on using the Racket FFI. If you haven&rsquo;t read +part 1 yet, you can find it +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a>. +<span style="font-weight: bold">Update:</span> part 3 is also now available +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.</p> +<!--more--> + +<p>First, here&rsquo;s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">bitmap magic</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_create</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Bitmap -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a helper for displaying the bitmap</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<h1><a name="(part._.Dashes_and_array_arguments)"></a>Dashes and array arguments</h1> + +<p>To start off, let&rsquo;s look at another C example from the Cairo +<a href="https://www.cairographics.org/samples/">samples page</a>. +This time we will look at the "dash" example, which has a +use of an input array:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">sizeof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">)</span><span class="RktMeta">/sizeof</span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktVal">0</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">50.0</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">25.6</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_rel_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">102.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_curve_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The most interesting function here is <span class="stt">cairo_set_dash</span>, which takes an +array argument. The only other new functions are <span class="stt">cairo_rel_line_to</span> +and <span class="stt">cairo_curve_to</span> which have very straightforward C types:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-rel-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_rel_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-curve-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_curve_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Meanwhile, the C type signature for <span class="stt">cairo_set_dash</span> from the Cairo +docs looks like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Something to note about the arguments is that <span class="stt">num_dashes</span> +encodes the length of the array <span class="stt">dashes</span>. This will come up later when +we want to make the C type for this function more convenient.</p> + +<p>On the Racket side, it&rsquo;s natural to represent the array of dashes as either a +list or <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/vectors.html#%28tech._vector%29"><span class="techinside">vector</span></a> of numbers. Given that, a fairly literal translation of +the C type above might look like the following:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This type includes a type constructor we haven&rsquo;t seen yet: <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span>. +This is a so-called <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28tech._custom._function._type%29"><span class="techinside">custom function type</span></a> that has special meaning +inside of a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.</p> + +<p>Since we only want to provide a list from the Racket side to C, we&rsquo;ll use +the <span class="RktWrap"><span class="RktSym">i</span></span> input mode. We can then call the function like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">4</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Note that because of how we defined the type of <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it&rsquo;s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.</p> + +<p>Luckily, the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type constructor actually lets you do this with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">name</span><span class="stt"> </span><span class="RktSym">:</span><span class="stt"> </span><span class="RktSym">type</span><span class="RktPn">)</span></span> syntax for naming arguments in combination with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">type</span><span class="stt"> </span><span class="RktVar">=</span><span class="stt"> </span><span class="RktSym">expr</span><span class="RktPn">)</span></span> syntax for supplying computed arguments:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">name this argument for later uses in the type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a computed argument position</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3d%29%29">=</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>When a computed argument is specified with a <span class="RktWrap"><span class="RktVar">=</span></span>, it&rsquo;s not necessary to provide +the argument on the Racket side. So <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> is now an arity 3 +function that can be called like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This means we&rsquo;ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it&rsquo;s also possible to use Racket vectors instead of lists by using +the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__vector%29%29">_vector</a></span></span> type constructor.</p> + +<p>Putting it all together, we can reproduce the dashes example like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">25.6</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-rel-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-1</span>02.4</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-curve-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Result_arguments_and_.C_structs)"></a>Result arguments and C structs</h1> + +<p>For some more advanced FFI hacking, let&rsquo;s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Our challenge is to make a Racket function that takes a string (let&rsquo;s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let&rsquo;s just scale the +text to fit the width and assume the height will be okay.</p> + +<p>To implement the key step of measuring the text size, we can use the +<a href="https://www.cairographics.org/manual/cairo-text.html#cairo-text-extents"><span class="stt">cairo_text_extents</span></a> +function. Its type signature is as follows:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">char</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*utf8,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*extents</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The interesting part of this signature is that +<a href="https://www.cairographics.org/manual/cairo-cairo-scaled-font-t.html#cairo-text-extents-t"><span class="stt">cairo_text_extents_t</span></a> +is a struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">Cairo</span><span class="hspace">&nbsp;</span><span class="RktCmt">docs</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">height</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>We haven&rsquo;t yet seen how to handle C structs with the FFI, but it&rsquo;s not +too tricky. Support for C structs comes built-in and will look familiar +if you&rsquo;re used to Racket structs. We can directly translate the documented +definition above into a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span> declaration:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the leading underscore is mandatory</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">width</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">height</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">x-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">pointer to struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">allows NULL pointer</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer/null</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr></tbody></table></div> + +<p>Along with functions that look like regular Racket struct operations:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a struct constructor</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:make-cairo_text_extents_t&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field selector</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:cairo_text_extents_t-width&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a predicate for the struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t?</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:^TYPE?&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field mutation function</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">set-cairo_text_extents_t-width!</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:set-cairo_text_extents_t-width!&gt;</span></p></td></tr></tbody></table></div> + +<p>With the struct type defined, it&rsquo;s easy to come up with a rudimentary +interface for <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>This style of programming feels awfully imperative though. Since we&rsquo;re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span> FFI wrapper +by combining named arguments, a new <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> type constructor, +and a neat feature of <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> that lets you customize +the return result:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">named args and _ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">ext</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the return result of the C function</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">custom return result for the wrapper</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">ext</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> constructor works like the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span> constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the <span class="RktWrap"><span class="RktSym">o</span></span> mode to <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span>. +In output mode, this type will automatically allocate a new instance of the type +(using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._malloc%29%29">malloc</a></span></span> function) and arrange for it to be passed in as +a pointer.</p> + +<p>The strangest part of this example is that there are now two uses of the +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span></span> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.</p> + +<p>Using this new version of the wrapper is much simpler:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>With that in hand, it&rsquo;s pretty easy to write the function we set out to write:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-show-text</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_show_text</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-scale</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">String -&gt; Void</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">draws a string scaled horizontally</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">20</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">2.0</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._-%29%29"><span class="nobreak">-</span></a></span><span class="hspace">&nbsp;</span><span class="RktVal">256.0</span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym">width</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-scale</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-show-text</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>And to conclude part 2 of this tutorial, here&rsquo;s an example use +of the new <span class="RktWrap"><span class="RktSym">fit-text</span></span> function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktVal">"Saluton, Mondo / Hallo, mundo"</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + + Tutorial: Using Racket's FFI + http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/?utm_source=Racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-27-tutorial-using-racket-s-ffi + Mon, 27 Jun 2016 16:22:11 UT + Asumu Takikawa + +<p><span style="font-weight: bold">Update:</span> this post is now part of a series. Part 2 is +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a> +and part 3 is +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>I&rsquo;ve seen several people ask for a tutorial on Racket&rsquo;s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +<a href="http://docs.racket-lang.org/foreign/index.html">documentation</a> +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.</p> + +<p>With that in mind, this blog post will provide a step-by-step tutorial +for Racket&rsquo;s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.</p> +<!--more--> + +<p>Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop <a href="http://www.ccs.neu.edu/racket/pubs/scheme04-bo.pdf">paper</a> +that you can read if you&rsquo;re curious about the design.</p> + +<p>The tutorial will focus on using the <a href="https://www.cairographics.org/">Cairo</a> +graphics library, mainly because it comes bundled with Racket.</p> + +<p>To start, let&rsquo;s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo&rsquo;s +<a href="https://www.cairographics.org/samples/">samples page</a>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">30.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">CAIRO_LINE_CAP_ROUND</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This uses the Racket drawing library <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/draw/index.html"><span class="RktSym">racket/draw</span></a></span> to construct +a bitmap object that we&rsquo;ll draw on. The <span class="RktWrap"><span class="RktSym">get-handle</span></span> method just extracts a +low-level Cairo surface value that we can use.</p> + +<p>NB: these code snippets don&rsquo;t come with a <span class="stt">#lang</span> declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy &amp; paste the snippets into your REPL.</p> + +<p>Our first real step is to import the FFI itself:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>As the module name suggests, the FFI is <span class="emph">unsafe</span> and can cause your Racket process +to segfault. If you&rsquo;re following along in DrRacket, you will want to save your file +frequently.</p> + +<p>Next, we can load the Cairo library to obtain a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28tech._foreign._library._value%29"><span class="techinside">foreign-library value</span></a>, which +is a handle that we use to access C values and functions:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Since Cairo has already been loaded by the Racket process because of the +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/gui/index.html"><span class="RktSym">racket/gui</span></a></span> import earlier, we can supply <span class="RktWrap"><span class="RktVal">#f</span></span> here as an +argument to <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span>. Normally you supply the name of a shared +library file such as <span class="RktWrap"><span class="RktVal">"libcairo"</span></span>:</p> + +<div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"libcairo"</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"2"</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></p></div> + +<p>The last list argument specifies the accepted versions (<span class="RktWrap"><span class="RktVal">#f</span></span> allows +a version-less library). For this post, those details aren&rsquo;t important but +see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span> if you&rsquo;re curious.</p> + +<h1><a name="(part._.Extracting_functions)"></a>Extracting functions</h1> + +<p>Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> function. The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> +function takes three arguments:</p> + +<ul> + <li> + <p>The name of the value as a string (or symbol or bytestring)</p></li> + <li> + <p>a foreign library value, and</p></li> + <li> + <p>a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/types.html#%28tech._c._type%29"><span class="techinside">C type</span></a>, which is a type description that tells +the FFI how to marshall between Racket and C.</p></li></ul> + +<p>C types are a crucial concept for the FFI. They range from relatively +simple types like <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> to more complicated +type constructors such as <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span>. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span></span> with two functions that +handle marshalling between C and Racket code.</p> + +<p>To make progress with our Cairo code, we need to create a drawing context from +the surface object <span class="RktWrap"><span class="RktSym">bt-surface</span></span> that we defined a while ago. The +relevant function in the Cairo docs is +<a href="https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-create"><span class="stt">cairo_create</span></a>, +which has the following type signature:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">NB:</span><span class="hspace">&nbsp;</span><span class="RktCmt">this</span><span class="hspace">&nbsp;</span><span class="RktCmt">is</span><span class="hspace">&nbsp;</span><span class="RktCmt">C</span><span class="hspace">&nbsp;</span><span class="RktCmt">code</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_create</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_surface_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*target</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a <span class="stt">cairo_surface_t</span> and +returns a pointer to a <span class="stt">cairo_t</span>. Let&rsquo;s start with a very simple C type +that matches up with this behavior: </div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span></p></div></div> + +<div class="SIntrapara">This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type.</div> + +<p>The following definition shows how to use this type to obtain a foreign +function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Then we can use <span class="RktWrap"><span class="RktSym">cairo-create</span></span> as an ordinary racket function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">ctx</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Interlude__more_type_safety)"></a>Interlude: more type safety</h1> + +<p>Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> types, it is easy +to accidentally misuse the function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">You may not want to actually run this</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a cairo_t is not a cairo_surface_t</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>To prevent such bad uses, it is good practice to use <span class="emph">tagged</span> pointer types +using <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>. Here are two example definitions that +correspond to the <span class="stt">cairo_t</span> and <span class="stt">cairo_surface_t</span> types from earlier:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">The leading underscores are mandatory</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>We can then redefine <span class="RktWrap"><span class="RktSym">cairo-create</span></span> with a better type, which will +prevent ill-typed calls:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">cairo_surface_t-&gt;C: argument is not non-null</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">`cairo_surface_t' pointer</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktErr">argument: #&lt;cpointer:cairo_t&gt;</span></p></td></tr></tbody></table></div> + +<p>Unfortunately our old definition of <span class="RktWrap"><span class="RktSym">ctx</span></span> doesn&rsquo;t have this tag:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#f</span></p></td></tr></tbody></table></div> + +<p>Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it&rsquo;s also possible to update +existing pointers with a tag like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-push-tag%21%29%29">cpointer-push-tag!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#t</span></p></td></tr></tbody></table></div> + +<p>Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).</p> + +<h1><a name="(part._.Macros_for_reducing_boilerplate)"></a>Macros for reducing boilerplate</h1> + +<p>Now let&rsquo;s start building the FFI bindings for the functions in the Cairo sample. +First, let&rsquo;s go ahead and look at all of the types for the sample functions +from the C API docs:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_cap_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">line_cap</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Starting with <span class="stt">cairo_move_to</span>, we can set up a definition like we did +with <span class="stt">cairo_create</span> before:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">"cairo_move_to"</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span> library that help reduce the +verbosity. Here&rsquo;s an alternative definition of <span class="RktWrap"><span class="RktSym">cairo-move-to</span></span> +using the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form from +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>As you can see, the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +<span class="RktWrap"><span class="RktSym">cairo_move_to</span></span>) you also don&rsquo;t need to supply the C name either.</p> + +<p>The definitions for <span class="stt">cairo_line_to</span>, <span class="stt">cairo_set_line_width</span>, and +<span class="stt">cairo_stroke</span> aren&rsquo;t very interesting, so I&rsquo;ll just include them +below without comment:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="stt">cairo_set_line_cap</span> case is more interesting because the type +<span class="stt">cairo_line_cap_t</span> is a C enumeration type. Racket&rsquo;s FFI comes with +convenience forms for defining enumeration types&#8212; + <wbr />though it&rsquo;s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here&rsquo;s a quote from the Barzilay and Orlovsky +paper on that:</p> + +<blockquote class="SubFlow"> + <p>Our design follows a simple principle: keep C-level +functionality to a minimum.</p></blockquote> + +<p>and specifically about enumerations:</p> + +<blockquote class="SubFlow"> + <p>For example, the C level part of our interface does not commit to a +specific implementation for enumerations &#8212; it simply exposes C integers.</p></blockquote> + +<p>To define an enumeration, we can use the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the <span class="stt">cairo_line_cap_t</span> type, it suffices to +just supply the cases as a list of symbols:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It&rsquo;s also possible to specify how the symbols map to integers more precisely +(see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> for those details).</p> + +<p>Given this type, we can specify the type for the line cap function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<h1><a name="(part._.Putting_it_all_together)"></a>Putting it all together</h1> + +<p>Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">30.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-cap</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">round</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object <span class="RktWrap"><span class="RktSym">bt</span></span>. To see the +results of what we drew, we can just evaluate <span class="RktWrap"><span class="RktSym">bt</span></span> at the REPL. But it&rsquo;s +a little nicer if we use the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> library to draw a frame around +it to distinguish it from the background:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>And we&rsquo;re done! Of course, there is a lot more to the FFI. For example, I haven&rsquo;t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I&rsquo;m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!</p> \ No newline at end of file diff --git "a/blog/feeds/R\314\214-JIT.atom.xml" "b/blog/feeds/R\314\214-JIT.atom.xml" new file mode 100644 index 00000000..43d79c73 --- /dev/null +++ "b/blog/feeds/R\314\214-JIT.atom.xml" @@ -0,0 +1,16 @@ + + + PRL Blog: Posts tagged 'Ř JIT' + + + urn:http-prl-ccs-neu-edu:-blog-tags-R-CC-8C-JIT-html + 2022-02-16T00:00:42Z + + [Ř Overview I (cross-post)](https://www.o1o.ch/lab/entry/1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc.html) + + urn:http-prl-ccs-neu-edu:-blog-2022-02-16-r-CC-8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html + 2022-02-16T00:00:42Z + 2022-02-16T00:00:42Z + + Olivier Flückiger + \ No newline at end of file diff --git "a/blog/feeds/R\314\214-JIT.rss.xml" "b/blog/feeds/R\314\214-JIT.rss.xml" new file mode 100644 index 00000000..75989e19 --- /dev/null +++ "b/blog/feeds/R\314\214-JIT.rss.xml" @@ -0,0 +1,16 @@ + + + + PRL Blog: Posts tagged 'Ř JIT' + PRL Blog: Posts tagged 'Ř JIT' + http://prl.ccs.neu.edu/blog/tags/R%CC%8C-JIT.html + Wed, 16 Feb 2022 00:00:42 UT + Wed, 16 Feb 2022 00:00:42 UT + 1800 + + [Ř Overview I (cross-post)](https://www.o1o.ch/lab/entry/1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc.html) + http://prl.ccs.neu.edu/blog/2022/02/16/-r%CC%8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/?utm_source=R%CC%8C-JIT&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-02-16-r-CC-8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html + Wed, 16 Feb 2022 00:00:42 UT + Olivier Flückiger + \ No newline at end of file diff --git a/blog/feeds/Scribble.atom.xml b/blog/feeds/Scribble.atom.xml new file mode 100644 index 00000000..72dc7abc --- /dev/null +++ b/blog/feeds/Scribble.atom.xml @@ -0,0 +1,756 @@ + + + PRL Blog: Posts tagged 'Scribble' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Scribble-html + 2019-02-17T16:20:50Z + + Writing a paper with Scribble + + urn:http-prl-ccs-neu-edu:-blog-2019-02-17-writing-a-paper-with-scribble + 2019-02-17T16:20:50Z + 2019-02-17T16:20:50Z + + Ben Greenman + +<p>This post explains how to get started using Scribble to write a research paper.</p> +<!-- more--> + +<hr /> + +<blockquote> + <p>This post was written using <a href="http://download.racket-lang.org/all-versions.html">Racket 7.1</a> and <a href="https://github.com/racket/scribble/releases/tag/v7.1">Scribble 1.29</a></p></blockquote> + +<p>Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:</p> + +<ul> + <li>the paper can import definitions from the research, keeping a single point of control;</li> + <li>the language&rsquo;s functional abstractions can help manage the writing;</li> + <li>the language&rsquo;s drawing and/or plotting libraries can replace <a href="https://ctan.org/pkg/pgf?lang=en">TikZ</a>;</li> + <li>and you can write unit tests to validate the claims made in the paper.</li></ul> + +<p>Scribble, <a href="http://docs.racket-lang.org/scribble/index.html">the Racket documentation tool</a>, comes with a to-LaTeX compiler and a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> library tailored to the new <a href="https://ctan.org/pkg/acmart?lang=en">ACM paper format</a>. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short &ldquo;getting started&rdquo; guide and links to some larger examples.</p> + +<blockquote> + <p>For a Scribble tutorial, see the links in: <a href="/blog/2017/05/23/building-a-website-with-scribble/index.html">Building a Website with Scribble</a></p></blockquote> + +<h2 id="getting-started-with-">Getting started with <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a></h2> + +<p>The first line of a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> document sets the formatting options (similar to a LaTeX file using <code>acmart.cls</code>). For example, the <a href="https://conf.researchr.org/track/gpce-2018/gpce-2018#Call-for-Papers">GPCE 2018 call for papers</a> asks for anonymized <code>sigplan</code>-format submissions with line numbers and 10 point font. The proper Scribble incantation is:</p> + +<pre><code>#lang scribble/acmart @sigplan @anonymous @review @10pt</code></pre> + +<p>Next, you may want to import some definitions. If we have a file <code>references.rkt</code> (see below for a definition), we can import it as follows:</p> + +<pre><code>@require{references.rkt}</code></pre> + +<p>The third main ingredient is the title and author information:</p> + +<pre><code>@(define neu (affiliation #:institution "Northeastern University")) +@(define anon (email "anon@anon.net")) + +@title{Writing a paper with Scribble} +@author[#:affiliation neu #:email anon]{Ben Greenman} + +@; optional: set the author names in the page headers +@elem[#:style "Sshortauthors"]{B. Greenman}</code></pre> + +<p>The paper is now ready to be written. You can forge ahead with a new <a href="http://docs.racket-lang.org/scribble/base.html#%28def._%28%28lib._scribble%2Fbase..rkt%29._section%29%29">section</a> and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and <a href="http://docs.racket-lang.org/scribble/base.html#%28form._%28%28lib._scribble%2Fbase..rkt%29._include-section%29%29">import</a> content from other modules:</p> + +<pre><code>@include-abstract{abstract.scrbl} +@include-section{introduction.scrbl}</code></pre> + +<p>Finally, the main page is a good place to <a href="https://docs.racket-lang.org/scriblib/autobib.html">generate the bibliography</a>. Assuming this document imports a file like the <code>references.rkt</code> below, this expression inserts a bibliography titled &ldquo;References&rdquo;:</p> + +<pre><code>@generate-bibliography[#:sec-title "References"]</code></pre> + +<p>To build the document, invoke <code>scribble</code> on the command-line with the <code>--pdf</code> or <code>--latex</code> options:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl</code></pre> + +<p>If all goes well, this command generates a <code>FILE.pdf</code> with properly-linked cross references.</p> + +<h3 id="auxiliary-files">Auxiliary Files</h3> + +<p>If you save the code above to a file <code>example.scrbl</code> and save the files below in the same directory, then you should be able to build an <code>example.pdf</code>.</p> + +<p>These files are available in a slightly different format at this link:</p> + +<ul> + <li><a href="https://gitlab.com/bengreenman/scribble-acmart-example">https://gitlab.com/bengreenman/scribble-acmart-example</a></li></ul> + +<h4 id="referencesrkt"><code>references.rkt</code></h4> + +<pre><code>#lang racket/base + +(provide + ~cite citet generate-bibliography + fbf-icfp-2009) + +(require + scriblib/autobib) + +(define-cite ~cite citet generate-bibliography + #:style author+date-square-bracket-style) + +(define icfp "ICFP") + +(define fbf-icfp-2009 + (make-bib + #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools" + #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler") + #:location (proceedings-location icfp #:pages '(109 120)) + #:date 2017))</code></pre> + +<h4 id="abstractscrbl"><code>abstract.scrbl</code></h4> + +<pre><code>#lang scribble/acmart + +A simple Scribble document.</code></pre> + +<h4 id="introductionscrbl"><code>introduction.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require{references.rkt} + +@; start with `title` instead of `section`, because importing via +@; `include-section` shifts all title/section/subsections down one level +@title{Introduction} + +Scribble creates a connection between a stand-alone document and the artifact +it describes@~cite[fbf-icfp-2009].</code></pre> + +<h3 id="q-how-to-debug-scribble-error-messages">Q. How to debug Scribble error messages?</h3> + +<p>If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.</p> + +<p>As a compile-time example, adding <code>@ foo</code> to a document produces the message <code>unexpected whitespace after @</code> and you can either delete the whitespace or change the <code>@</code> to <code>@"@"</code> for a literal <code>@</code>-sign.</p> + +<p>As a run-time example, adding <code>@(+ 2 2)</code> produces this message:</p> + +<pre><code>not valid in document body (need a pre-part for decode) in: 4</code></pre> + +<p>One fix is to convert <code>4</code> to a string, as in <code>@~a[(+ 2 2)]</code>.</p> + +<p>But if something goes wrong when Scribble renders a generated document to PDF, the default error output is <strong>not</strong> likely to help. For example, adding <code>@elem[#:style "oops"]</code> to a document produces a giant message:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl +[[ ... 84K of output ... ]] +Output written on example.pdf (1 page, 277876 bytes). +PDF statistics: + 53 PDF objects out of 1000 (max. 8388607) + 37 compressed objects within 1 object stream + 7 named destinations out of 1000 (max. 500000) + 36877 words of extra memory for PDF output out of 42996 (max. 10000000) + +run-pdflatex: got error exit code + context...: + [[ ... 17 more lines ... ]]</code></pre> + +<p>The best way to debug these messages is to <strong>ignore them</strong> and use a LaTeX compiler directly. For the &ldquo;oops&rdquo; mistake, LaTeX stops at the undefined control sequence &mdash; giving a hint about how to find the problem:</p> + +<pre><code>$ raco scribble --latex FILE.scrbl +$ pdflatex FILE.tex +[[ ... 12KB of output ... ]] +! Undefined control sequence. +l.549 \oops + {} +? </code></pre> + +<h3 id="q-how-to-add-a-latex-style-file">Q. How to add a LaTeX style file?</h3> + +<p>To add extra LaTeX code to the final document, create a new file and include it with the <code>++style</code> command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).</p> + +<pre><code>$ raco scribble ++style style.tex --pdf FILE.scrbl</code></pre> + +<p>Here is an example style file.</p> + +<h4 id="styletex"><code>style.tex</code></h4> + +<pre><code>\settopmatter{printfolios=true,printccs=true,printacmref=true} +% add page numbers etc. + +\overfullrule=1mm +% draw a black rectangle near lines that overflow the margin</code></pre> + +<p>Another way to add extra LaTeX code is to add a <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Flatex-properties..rkt%29._tex-addition%29%29"><code>tex-addition</code></a> style property to the main title. This second approach makes it easy to include more than one file:</p> + +<pre><code>#lang scribble/acmart + +@require[ + (only-in scribble/core make-style) + (only-in scribble/latex-properties make-tex-addition)] + +@(define extra-style-files + (list (make-tex-addition "style.tex"))) + +@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble} + +@; ....</code></pre> + +<h3 id="q-how-to-make-a-figure">Q. How to make a figure?</h3> + +<p>Use the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library to add figures to a document.</p> + +<pre><code>@require[pict scriblib/figure] +@figure[ + "fig:fish" @; figure tag, see `figure-ref` + @elem{A Standard Fish} @; figure caption, appears below the content + @elem{fish = @(standard-fish 90 40)}] @; content</code></pre> + +<p>The content of a figure can be almost anything that would work in the toplevel of the document.</p> + +<h3 id="q-how-to-include-extra-files-pictures-latex">Q. How to include extra files (pictures, LaTeX)?</h3> + +<p>The <code>++extra</code> command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.</p> + +<p>For example, if a document includes the content of an external LaTeX file:</p> + +<pre><code>@elem[#:style "input"]{inline-this.tex}</code></pre> + +<p>then make sure to build the document with a command like this:</p> + +<pre><code>$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl</code></pre> + +<h4 id="inline-thistex"><code>inline-this.tex</code></h4> + +<pre><code>% Raw LaTeX allowed here +$\lambda x.\, x$</code></pre> + +<h3 id="q-what-about-in-line-latex">Q. What about in-line LaTeX?</h3> + +<p>An <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Fcore..rkt%29._element%29%29">element</a> with the <a href="https://docs.racket-lang.org/scribble/core.html#%28idx._%28gentag._60._%28lib._scribblings%2Fscribble%2Fscribble..scrbl%29%29%29"><code>'exact-chars</code></a> <a href="https://docs.racket-lang.org/scribble/core.html#%28tech._style._property%29">style property</a> renders directly to LaTeX.</p> + +<pre><code>@(define (exact . stuff) + @; the style name "relax" puts a `\relax` no-op in front of the stuff + (make-element (make-style "relax" '(exact-chars)) stuff)) + +@exact|{$\lambda x.\, x$}| +@; ==&gt; \relax{$\lambda x.\, x$} + +@(define ($ . math-stuff) + (apply exact (list "$" math-stuff "$"))) + +@${\lambda x.\, x} +@; ==&gt; \relax{$\lambda x.\, x$}</code></pre> + +<h2 id="creating-a-httpdocsracket-langorgguidemodulesyntaxhtml28parthash-lang29lang-for-a-paper">Creating a <a href="http://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29">#lang</a> for a paper</h2> + +<p>For a Scribble document that is split across multiple files, it can be helpful to make a <code>#lang</code> that <a href="http://blog.racket-lang.org/2017/03/languages-as-dotfiles.html">provides a common environment</a>. Instead of starting each file with a <code>require</code>, e.g.:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require["references.rkt" "helper-functions.rkt" scriblib/figure] + +....</code></pre> + +<p>files can start with a name that describes their common purpose:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang conference-2018-submission + +....</code></pre> + +<p>As a bonus, if the language is defined as a package then the Scribble document can use Racket&rsquo;s dependency management tools:</p> + +<pre><code># to install the paper and interactively install dependencies: +$ cd conference-2018-submission; +$ raco pkg install + +# To check that the paper builds with no dependency issues: +$ raco setup --check-pkg-deps conference-2018-submission + +# To run all unit tests +$ raco test -c conference-2018-submission</code></pre> + +<p>To create a package and language:</p> + +<ol> + <li>Move the Scribble document to a directory with the language name, i.e., <code>conference-2018-submission/</code></li> + <li>Write a simple <code>info.rkt</code> to configure the package</li> + <li>Create a normal Racket module that exports the common environment</li> + <li>Create a <code>conference-2018-submission/lang/reader.rkt</code> module</li></ol> + +<p>Details below. For a full example, visit:</p> + +<ul> + <li><a href="https://gitlab.com/bennn/scribble-acmart-example">https://gitlab.com/bennn/scribble-acmart-example</a></li></ul> + +<hr /> + +<h4 id="conference-2018-submissioninforkt"><code>conference-2018-submission/info.rkt</code></h4> + +<p>This file defines the basic metadata for a package. For more about <code>info.rkt</code>, see: <a href="http://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html">Tutorial: Creating a Package</a>.</p> + +<pre><code>#lang info +(define collection "conference-2018-submission") +(define deps '("base" "scribble-lib" "at-exp-lib")) +(define build-deps '("racket-doc" "scribble-doc")) +(define pkg-desc "Paper for Conference 2018") +(define version "0.1")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionmainrkt"><code>conference-2018-submission/main.rkt</code></h4> + +<p>This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> language, the file &ldquo;references.rkt&rdquo;, and the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library.</p> + +<pre><code>#lang racket/base + +(provide + (all-from-out + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")) + +(require + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionlangreaderrkt"><code>conference-2018-submission/lang/reader.rkt</code></h4> + +<p>This file: (1) tells Racket to use the Scribble reader on <code>#lang conference-2018-submission</code> modules, and (2) wraps the result of such modules in a shape that Scribble expects.</p> + +<pre><code>#lang s-exp scribble/base/reader +conference-2018-submission +#:wrapper1 (lambda (t) (cons 'doc (t)))</code></pre> + +<h2 id="links-to-example-documents">Links to Example Documents</h2> + +<p>These documents use the <code>#lang</code> approach to writing a paper with Scribble. Check their <code>main.rkt</code> for example formatting functions and unit tests, and check the <code>.scrbl</code> files to see how the ideas above look in a larger document.</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018">https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018</a></li> + <li><a href="https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018">https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018</a></li></ul> + +<p>Finally, this repository provides a tool to start a new Scribble document:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gtp-paper">https://pkgd.racket-lang.org/pkgn/package/gtp-paper</a></li></ul> + +<h2 id="further-reading">Further Reading</h2> + +<ul> + <li><a href="https://project.inria.fr/coqexchange/checking-machine-checked-proofs/">Checking Machine-Checked Proofs</a></li></ul> + + Building a Website with Scribble + + urn:http-prl-ccs-neu-edu:-blog-2017-05-23-building-a-website-with-scribble + 2017-05-23T01:53:13Z + 2017-05-23T01:53:13Z + + Ben Greenman + +<p>The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!</p> +<!-- more--> + +<h2 id="the-story-so-far">The Story so Far</h2> + +<p>Last Fall, I took a flight to Chicago (on my way to <a href="http://con.racket-lang.org/2016/">RacketCon 2016</a>). When I landed, there was a new message in my inbox:</p> + +<pre><code> Subject: Web Page + Date: 2016-09-15 + + You have been nominated webmaster by public acclamation. Congratulations!</code></pre> + +<p>Emboldened by the trust of my people, I promptly converted the PRL website from Racket-generating-HTML to the fine <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> preprocessor language (commit <a href="https://github.com/nuprl/website/commit/a0600d32fec4bd70c5530b2717aec32979d634f7"><code>a0600d</code></a>) This bold action polarized the community.</p> + +<blockquote> + <p>I can&rsquo;t read the source anymore! Is this really an improvement?</p></blockquote> + +<p>Fear not, citizens. The switch to <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> was the right choice, and you too can learn to read the source code.</p> + +<h2 id="how-to-read-scribblehtml-programs">How to Read <code>scribble/html</code> Programs</h2> + +<h3 id="basics">Basics</h3> + +<p>Scribble is a language for writing Racket documentation. The key innovation in Scribble is the <em>@-expression</em> (read: &ldquo;at expression&rdquo;). The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language combines @-expression syntax with functions that generate HTML.</p> + +<h4 id="-syntax">@-syntax</h4> + +<p><a href="http://www.greghendershott.com/2015/08/at-expressions.html">Greg Hendershott</a> and the <a href="http://docs.racket-lang.org/scribble/reader.html">Scribble Documentation</a> explain @-expressions properly. Here&rsquo;s a short tutorial (Part 1 of 2, &ldquo;the basics&rdquo;):</p> + +<ul> + <li>Scribble programs start in &ldquo;text mode&rdquo;. Every character you type goes straight to the document you are building.</li> + <li>The @-sign toggles to &ldquo;Racket mode&rdquo; for the next expression. In Racket mode, the characters you type will be evaluated as a Racket program to produce part of the document.</li></ul> + +<p><em>Examples:</em> Evaluating <code>"Hello Dave"</code> puts &ldquo;Hello Dave&rdquo; in your document. Evaluating <code>"Hello @Dave"</code> puts &ldquo;Hello ???&rdquo; in your document, where "???" is the value of the variable <code>Dave</code>. Finally if <code>Dave</code> is the name of a function, then <code>"Hello @(Dave)"</code> calls the <code>Dave</code> function with zero arguments and puts whatever it returns into your document.</p> + +<p>To make it easy to interleave text, function calls, and code, Scribble discriminates between 4 kinds of parentheses when they follow an @-sign (Part 2 of 2, &ldquo;the parens&rdquo;):</p> + +<ul> + <li><code>@(f A B)</code> is just like the function call <code>(f A B)</code> in Racket</li> + <li><code>@f[A B]</code> is the same as <code>@(f A B)</code>, but typically more useful because &hellip;</li> + <li><code>@f[A B]{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; to a list of words <code>w*</code>, then calls <code>f</code> just like <code>(apply f A B w*)</code></li> + <li><code>@f{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; and calls <code>f</code> with the results</li> + <li><code>@f|{....}|</code> is similar, but the <code>....</code> are in &ldquo;unescapable text mode&rdquo;</li></ul> + +<p>&ldquo;Unescapable text mode&rdquo; treats @-signs as text instead of toggling between modes.</p> + +<h4 id="generating-html">Generating HTML</h4> + +<p>The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language comes with functions that render HTML. These functions have the same name as the corresponding HTML tag.</p> + +<p>Example program:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this program prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>No surprises.</p> + +<p>One thing that <em>is</em> surprising is how <code>scribble/html</code> handles tag attributes. Every tag-rendering function accepts &ldquo;Racket mode&rdquo; arguments that specify an attribute name and attribute value.</p> + +<p>For example:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">[</span><span class="n">style:</span> <span class="s2">"color:red"</span><span class="p">]{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Hope the output looks familiar. The input syntax is strange, but that&rsquo;s what it is.</p> + +<p>Larger programs print larger webpages. Each page on the PRL website is HTML generated by one <code>scribble/html</code> program.</p> + +<h2 id="why-scribblehtml-is-an-improvement">Why <code>scribble/html</code> is an Improvement</h2> + +<p>Before <code>scribble/html</code>, the PRL website was implemented in <code>scribble/text</code>. A <code>scribble/text</code> program renders and prints text. There is no extra support for HTML.</p> + +<p>To compare, here&rsquo;s the start of the old homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/text</span> +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="s2">"templates.rkt"</span><span class="p">)</span> + +<span class="n">&lt;!DOCTYPE</span> <span class="n">html&gt;</span> +<span class="n">&lt;html</span> <span class="n">lang=</span><span class="s2">"en"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">header</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;body</span> <span class="n">id=</span><span class="s2">"pn-top"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">navbar</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;div</span> <span class="n">class=</span><span class="s2">"jumbotron"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And here is the start of the <code>scribble/html</code>&rsquo;d homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@require</span><span class="p">[</span><span class="s2">"templates.rkt"</span><span class="p">]</span> + +<span class="n">@doctype</span><span class="p">{</span><span class="n">html</span><span class="p">}</span> +<span class="n">@html</span><span class="p">[</span><span class="n">lang:</span> <span class="s2">"en"</span><span class="p">]{</span> + <span class="n">@header</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@body</span><span class="p">[</span><span class="n">id:</span> <span class="s2">"pn-top"</span><span class="p">]{</span> + <span class="n">@navbar</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@div</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"jumbotron"</span><span class="p">]{</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The pages look similar. The new one has more @-signs and parentheses, the old one has more <code>&lt;</code>-signs and quotes. If you were able to edit the old page, you should be able to edit the new page.</p> + +<p>The <strong>key improvement</strong> in the new page is that <strong>common mistakes are now compile-time errors</strong>.</p> + +<ul> + <li> + <p>Before, a typo like <code>&lt;hmtl&gt;</code> would generate an ugly webpage. After, a typo like <code>@hmtl</code> is a syntax error.</p></li> + <li> + <p>Before, a typo like <code>&lt;b&gt;....</code> with no closing tag would generate an ugly webpage. After, a typo like <code>@b{....</code> is a syntax error.</p></li></ul> + +<p>Both flavors of error message come with source-code line numbers. This is very very helpful.</p> + +<h3 id="small-improvements">Small Improvements</h3> + +<h4 id="1-more-functions">1. More Functions</h4> + +<p>Before, the <a href="http://prl.ccs.neu.edu/teaching.html">Teaching page</a> contained some interesting HTML for rendering vertical text (look for the word &ldquo;Semantics&rdquo; to see how this was used):</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"how-to-design-programs"</span><span class="p">&gt;</span>S<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>e<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>m<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>a<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>n<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>t<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>i<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>c<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>s<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>After, the same text is generated from a function call:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@span</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"how-to-design-programs"</span><span class="p">]{</span><span class="n">@vertical-text</span><span class="p">{</span><span class="n">Semantics</span><span class="p">}}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The <code>vertical-text</code> function is simple:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@require</span><span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._only-in))" style="color: inherit">only-in</a></span> <span class="n">racket/list</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span><span class="p">)]</span> + +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">vertical-text</span> <span class="o">.</span> <span class="n">str*</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-~3elist))" style="color: inherit">string-&gt;list</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._append*))" style="color: inherit">append*</a></span> <span class="n">str*</span><span class="p">))</span> <span class="p">(</span><span class="n">br</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="2-more-structure-less-boilerplate">2. More Structure, Less Boilerplate</h4> + +<p>Here&rsquo;s part of the old definition of &ldquo;Ben Greenman&rdquo; on the <a href="http://prl.ccs.neu.edu/people.html">People page</a>:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"row pn-person"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-row-eq-height"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-photo"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"img-wrapper"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">"img/ben_greenman.jpg"</span> <span class="na">title</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="na">alt</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="p">/&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-9"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-4 pn-contact"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"pn-name"</span><span class="p">&gt;</span>Ben Greenman<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + Advisor: Matthias Felleisen<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"mailto:types@"</span><span class="err">@"</span><span class="na">ccs</span><span class="err">.</span><span class="na">neu</span><span class="err">.</span><span class="na">edu</span><span class="err">"</span><span class="p">&gt;</span>types@"@"ccs.neu.edu<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"http://www.ccs.neu.edu/home/types"</span><span class="p">&gt;</span>www.ccs.neu.edu/home/types<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-muted col-md-offset-5"</span><span class="p">&gt;</span> + Joined 2014 + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-bio"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>I like constructions .... <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new definition uses a helper function with keyword arguments for each &ldquo;field&rdquo; of the person:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@person</span><span class="p">[</span><span class="kd">#:name</span> <span class="s2">"Ben Greenman"</span> + <span class="kd">#:title</span> <span class="s2">"Advisor: Matthias Felleisen"</span> + <span class="kd">#:e-mail</span> <span class="s2">"types@ccs.neu.edu"</span> + <span class="kd">#:website</span> <span class="s2">"http://ccs.neu.edu/home/types"</span> + <span class="kd">#:history</span> <span class="n">@list</span><span class="p">[</span><span class="s2">"Joined 2014"</span><span class="p">]</span> + <span class="kd">#:img</span> <span class="s2">"ben_greenman.jpg"</span><span class="p">]{</span> + <span class="n">I</span> <span class="n">like</span> <span class="n">constructions</span> <span class="n">....</span> +<span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="3-less-string-formatting">3. Less String-Formatting</h4> + +<p>Before, the code did a lot of string formatting (<a href="https://github.com/nuprl/website/commit/a0600d#diff-1921e33ce89be28dd277cf1c7880d1beL9">link</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/creatingunits.html#(form._((lib._racket/unit..rkt)._link))" style="color: inherit">link</a></span> <span class="n">url</span> <span class="n">body</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-append))" style="color: inherit">string-append</a></span> <span class="s2">"&lt;a href=</span><span class="se">\"</span><span class="s2">"</span> <span class="n">url</span> <span class="s2">"</span><span class="se">\"</span><span class="s2">&gt;"</span> <span class="n">body</span> <span class="s2">"&lt;/a&gt;"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new code has no need for such helper functions.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@a</span><span class="p">[</span><span class="n">href:</span> <span class="n">url</span> <span class="n">body</span><span class="p">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="bottom-line">Bottom Line</h4> + +<p>Scribble is a good language for making static HTML pages.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://docs.racket-lang.org/pollen/index.html">Pollen</a></li> + <li><a href="https://github.com/vishesh/racketscript">RacketScript</a></li> + <li>Other websites built using <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a>: (1) <a href="http://nanopass.org/">nanopass.github.io</a> (<a href="https://github.com/nanopass/nanopass.github.io">source code</a>), (2) <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> (<a href="https://github.com/nuprl/gtp">source code</a>).</li> + <li><a href="http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/">Notes from a Gradual Typing Across the Spectrum PI meeting</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/Scribble.rss.xml b/blog/feeds/Scribble.rss.xml new file mode 100644 index 00000000..0042679c --- /dev/null +++ b/blog/feeds/Scribble.rss.xml @@ -0,0 +1,754 @@ + + + + PRL Blog: Posts tagged 'Scribble' + PRL Blog: Posts tagged 'Scribble' + http://prl.ccs.neu.edu/blog/tags/Scribble.html + Sun, 17 Feb 2019 16:20:50 UT + Sun, 17 Feb 2019 16:20:50 UT + 1800 + + Writing a paper with Scribble + http://prl.ccs.neu.edu/blog/2019/02/17/writing-a-paper-with-scribble/?utm_source=Scribble&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-02-17-writing-a-paper-with-scribble + Sun, 17 Feb 2019 16:20:50 UT + Ben Greenman + +<p>This post explains how to get started using Scribble to write a research paper.</p> +<!-- more--> + +<hr /> + +<blockquote> + <p>This post was written using <a href="http://download.racket-lang.org/all-versions.html">Racket 7.1</a> and <a href="https://github.com/racket/scribble/releases/tag/v7.1">Scribble 1.29</a></p></blockquote> + +<p>Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:</p> + +<ul> + <li>the paper can import definitions from the research, keeping a single point of control;</li> + <li>the language&rsquo;s functional abstractions can help manage the writing;</li> + <li>the language&rsquo;s drawing and/or plotting libraries can replace <a href="https://ctan.org/pkg/pgf?lang=en">TikZ</a>;</li> + <li>and you can write unit tests to validate the claims made in the paper.</li></ul> + +<p>Scribble, <a href="http://docs.racket-lang.org/scribble/index.html">the Racket documentation tool</a>, comes with a to-LaTeX compiler and a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> library tailored to the new <a href="https://ctan.org/pkg/acmart?lang=en">ACM paper format</a>. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short &ldquo;getting started&rdquo; guide and links to some larger examples.</p> + +<blockquote> + <p>For a Scribble tutorial, see the links in: <a href="/blog/2017/05/23/building-a-website-with-scribble/index.html">Building a Website with Scribble</a></p></blockquote> + +<h2 id="getting-started-with-">Getting started with <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a></h2> + +<p>The first line of a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> document sets the formatting options (similar to a LaTeX file using <code>acmart.cls</code>). For example, the <a href="https://conf.researchr.org/track/gpce-2018/gpce-2018#Call-for-Papers">GPCE 2018 call for papers</a> asks for anonymized <code>sigplan</code>-format submissions with line numbers and 10 point font. The proper Scribble incantation is:</p> + +<pre><code>#lang scribble/acmart @sigplan @anonymous @review @10pt</code></pre> + +<p>Next, you may want to import some definitions. If we have a file <code>references.rkt</code> (see below for a definition), we can import it as follows:</p> + +<pre><code>@require{references.rkt}</code></pre> + +<p>The third main ingredient is the title and author information:</p> + +<pre><code>@(define neu (affiliation #:institution "Northeastern University")) +@(define anon (email "anon@anon.net")) + +@title{Writing a paper with Scribble} +@author[#:affiliation neu #:email anon]{Ben Greenman} + +@; optional: set the author names in the page headers +@elem[#:style "Sshortauthors"]{B. Greenman}</code></pre> + +<p>The paper is now ready to be written. You can forge ahead with a new <a href="http://docs.racket-lang.org/scribble/base.html#%28def._%28%28lib._scribble%2Fbase..rkt%29._section%29%29">section</a> and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and <a href="http://docs.racket-lang.org/scribble/base.html#%28form._%28%28lib._scribble%2Fbase..rkt%29._include-section%29%29">import</a> content from other modules:</p> + +<pre><code>@include-abstract{abstract.scrbl} +@include-section{introduction.scrbl}</code></pre> + +<p>Finally, the main page is a good place to <a href="https://docs.racket-lang.org/scriblib/autobib.html">generate the bibliography</a>. Assuming this document imports a file like the <code>references.rkt</code> below, this expression inserts a bibliography titled &ldquo;References&rdquo;:</p> + +<pre><code>@generate-bibliography[#:sec-title "References"]</code></pre> + +<p>To build the document, invoke <code>scribble</code> on the command-line with the <code>--pdf</code> or <code>--latex</code> options:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl</code></pre> + +<p>If all goes well, this command generates a <code>FILE.pdf</code> with properly-linked cross references.</p> + +<h3 id="auxiliary-files">Auxiliary Files</h3> + +<p>If you save the code above to a file <code>example.scrbl</code> and save the files below in the same directory, then you should be able to build an <code>example.pdf</code>.</p> + +<p>These files are available in a slightly different format at this link:</p> + +<ul> + <li><a href="https://gitlab.com/bengreenman/scribble-acmart-example">https://gitlab.com/bengreenman/scribble-acmart-example</a></li></ul> + +<h4 id="referencesrkt"><code>references.rkt</code></h4> + +<pre><code>#lang racket/base + +(provide + ~cite citet generate-bibliography + fbf-icfp-2009) + +(require + scriblib/autobib) + +(define-cite ~cite citet generate-bibliography + #:style author+date-square-bracket-style) + +(define icfp "ICFP") + +(define fbf-icfp-2009 + (make-bib + #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools" + #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler") + #:location (proceedings-location icfp #:pages '(109 120)) + #:date 2017))</code></pre> + +<h4 id="abstractscrbl"><code>abstract.scrbl</code></h4> + +<pre><code>#lang scribble/acmart + +A simple Scribble document.</code></pre> + +<h4 id="introductionscrbl"><code>introduction.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require{references.rkt} + +@; start with `title` instead of `section`, because importing via +@; `include-section` shifts all title/section/subsections down one level +@title{Introduction} + +Scribble creates a connection between a stand-alone document and the artifact +it describes@~cite[fbf-icfp-2009].</code></pre> + +<h3 id="q-how-to-debug-scribble-error-messages">Q. How to debug Scribble error messages?</h3> + +<p>If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.</p> + +<p>As a compile-time example, adding <code>@ foo</code> to a document produces the message <code>unexpected whitespace after @</code> and you can either delete the whitespace or change the <code>@</code> to <code>@"@"</code> for a literal <code>@</code>-sign.</p> + +<p>As a run-time example, adding <code>@(+ 2 2)</code> produces this message:</p> + +<pre><code>not valid in document body (need a pre-part for decode) in: 4</code></pre> + +<p>One fix is to convert <code>4</code> to a string, as in <code>@~a[(+ 2 2)]</code>.</p> + +<p>But if something goes wrong when Scribble renders a generated document to PDF, the default error output is <strong>not</strong> likely to help. For example, adding <code>@elem[#:style "oops"]</code> to a document produces a giant message:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl +[[ ... 84K of output ... ]] +Output written on example.pdf (1 page, 277876 bytes). +PDF statistics: + 53 PDF objects out of 1000 (max. 8388607) + 37 compressed objects within 1 object stream + 7 named destinations out of 1000 (max. 500000) + 36877 words of extra memory for PDF output out of 42996 (max. 10000000) + +run-pdflatex: got error exit code + context...: + [[ ... 17 more lines ... ]]</code></pre> + +<p>The best way to debug these messages is to <strong>ignore them</strong> and use a LaTeX compiler directly. For the &ldquo;oops&rdquo; mistake, LaTeX stops at the undefined control sequence &mdash; giving a hint about how to find the problem:</p> + +<pre><code>$ raco scribble --latex FILE.scrbl +$ pdflatex FILE.tex +[[ ... 12KB of output ... ]] +! Undefined control sequence. +l.549 \oops + {} +? </code></pre> + +<h3 id="q-how-to-add-a-latex-style-file">Q. How to add a LaTeX style file?</h3> + +<p>To add extra LaTeX code to the final document, create a new file and include it with the <code>++style</code> command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).</p> + +<pre><code>$ raco scribble ++style style.tex --pdf FILE.scrbl</code></pre> + +<p>Here is an example style file.</p> + +<h4 id="styletex"><code>style.tex</code></h4> + +<pre><code>\settopmatter{printfolios=true,printccs=true,printacmref=true} +% add page numbers etc. + +\overfullrule=1mm +% draw a black rectangle near lines that overflow the margin</code></pre> + +<p>Another way to add extra LaTeX code is to add a <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Flatex-properties..rkt%29._tex-addition%29%29"><code>tex-addition</code></a> style property to the main title. This second approach makes it easy to include more than one file:</p> + +<pre><code>#lang scribble/acmart + +@require[ + (only-in scribble/core make-style) + (only-in scribble/latex-properties make-tex-addition)] + +@(define extra-style-files + (list (make-tex-addition "style.tex"))) + +@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble} + +@; ....</code></pre> + +<h3 id="q-how-to-make-a-figure">Q. How to make a figure?</h3> + +<p>Use the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library to add figures to a document.</p> + +<pre><code>@require[pict scriblib/figure] +@figure[ + "fig:fish" @; figure tag, see `figure-ref` + @elem{A Standard Fish} @; figure caption, appears below the content + @elem{fish = @(standard-fish 90 40)}] @; content</code></pre> + +<p>The content of a figure can be almost anything that would work in the toplevel of the document.</p> + +<h3 id="q-how-to-include-extra-files-pictures-latex">Q. How to include extra files (pictures, LaTeX)?</h3> + +<p>The <code>++extra</code> command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.</p> + +<p>For example, if a document includes the content of an external LaTeX file:</p> + +<pre><code>@elem[#:style "input"]{inline-this.tex}</code></pre> + +<p>then make sure to build the document with a command like this:</p> + +<pre><code>$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl</code></pre> + +<h4 id="inline-thistex"><code>inline-this.tex</code></h4> + +<pre><code>% Raw LaTeX allowed here +$\lambda x.\, x$</code></pre> + +<h3 id="q-what-about-in-line-latex">Q. What about in-line LaTeX?</h3> + +<p>An <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Fcore..rkt%29._element%29%29">element</a> with the <a href="https://docs.racket-lang.org/scribble/core.html#%28idx._%28gentag._60._%28lib._scribblings%2Fscribble%2Fscribble..scrbl%29%29%29"><code>'exact-chars</code></a> <a href="https://docs.racket-lang.org/scribble/core.html#%28tech._style._property%29">style property</a> renders directly to LaTeX.</p> + +<pre><code>@(define (exact . stuff) + @; the style name "relax" puts a `\relax` no-op in front of the stuff + (make-element (make-style "relax" '(exact-chars)) stuff)) + +@exact|{$\lambda x.\, x$}| +@; ==&gt; \relax{$\lambda x.\, x$} + +@(define ($ . math-stuff) + (apply exact (list "$" math-stuff "$"))) + +@${\lambda x.\, x} +@; ==&gt; \relax{$\lambda x.\, x$}</code></pre> + +<h2 id="creating-a-httpdocsracket-langorgguidemodulesyntaxhtml28parthash-lang29lang-for-a-paper">Creating a <a href="http://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29">#lang</a> for a paper</h2> + +<p>For a Scribble document that is split across multiple files, it can be helpful to make a <code>#lang</code> that <a href="http://blog.racket-lang.org/2017/03/languages-as-dotfiles.html">provides a common environment</a>. Instead of starting each file with a <code>require</code>, e.g.:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require["references.rkt" "helper-functions.rkt" scriblib/figure] + +....</code></pre> + +<p>files can start with a name that describes their common purpose:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang conference-2018-submission + +....</code></pre> + +<p>As a bonus, if the language is defined as a package then the Scribble document can use Racket&rsquo;s dependency management tools:</p> + +<pre><code># to install the paper and interactively install dependencies: +$ cd conference-2018-submission; +$ raco pkg install + +# To check that the paper builds with no dependency issues: +$ raco setup --check-pkg-deps conference-2018-submission + +# To run all unit tests +$ raco test -c conference-2018-submission</code></pre> + +<p>To create a package and language:</p> + +<ol> + <li>Move the Scribble document to a directory with the language name, i.e., <code>conference-2018-submission/</code></li> + <li>Write a simple <code>info.rkt</code> to configure the package</li> + <li>Create a normal Racket module that exports the common environment</li> + <li>Create a <code>conference-2018-submission/lang/reader.rkt</code> module</li></ol> + +<p>Details below. For a full example, visit:</p> + +<ul> + <li><a href="https://gitlab.com/bennn/scribble-acmart-example">https://gitlab.com/bennn/scribble-acmart-example</a></li></ul> + +<hr /> + +<h4 id="conference-2018-submissioninforkt"><code>conference-2018-submission/info.rkt</code></h4> + +<p>This file defines the basic metadata for a package. For more about <code>info.rkt</code>, see: <a href="http://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html">Tutorial: Creating a Package</a>.</p> + +<pre><code>#lang info +(define collection "conference-2018-submission") +(define deps '("base" "scribble-lib" "at-exp-lib")) +(define build-deps '("racket-doc" "scribble-doc")) +(define pkg-desc "Paper for Conference 2018") +(define version "0.1")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionmainrkt"><code>conference-2018-submission/main.rkt</code></h4> + +<p>This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> language, the file &ldquo;references.rkt&rdquo;, and the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library.</p> + +<pre><code>#lang racket/base + +(provide + (all-from-out + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")) + +(require + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionlangreaderrkt"><code>conference-2018-submission/lang/reader.rkt</code></h4> + +<p>This file: (1) tells Racket to use the Scribble reader on <code>#lang conference-2018-submission</code> modules, and (2) wraps the result of such modules in a shape that Scribble expects.</p> + +<pre><code>#lang s-exp scribble/base/reader +conference-2018-submission +#:wrapper1 (lambda (t) (cons 'doc (t)))</code></pre> + +<h2 id="links-to-example-documents">Links to Example Documents</h2> + +<p>These documents use the <code>#lang</code> approach to writing a paper with Scribble. Check their <code>main.rkt</code> for example formatting functions and unit tests, and check the <code>.scrbl</code> files to see how the ideas above look in a larger document.</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018">https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018</a></li> + <li><a href="https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018">https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018</a></li></ul> + +<p>Finally, this repository provides a tool to start a new Scribble document:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gtp-paper">https://pkgd.racket-lang.org/pkgn/package/gtp-paper</a></li></ul> + +<h2 id="further-reading">Further Reading</h2> + +<ul> + <li><a href="https://project.inria.fr/coqexchange/checking-machine-checked-proofs/">Checking Machine-Checked Proofs</a></li></ul> + + Building a Website with Scribble + http://prl.ccs.neu.edu/blog/2017/05/23/building-a-website-with-scribble/?utm_source=Scribble&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-23-building-a-website-with-scribble + Tue, 23 May 2017 01:53:13 UT + Ben Greenman + +<p>The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!</p> +<!-- more--> + +<h2 id="the-story-so-far">The Story so Far</h2> + +<p>Last Fall, I took a flight to Chicago (on my way to <a href="http://con.racket-lang.org/2016/">RacketCon 2016</a>). When I landed, there was a new message in my inbox:</p> + +<pre><code> Subject: Web Page + Date: 2016-09-15 + + You have been nominated webmaster by public acclamation. Congratulations!</code></pre> + +<p>Emboldened by the trust of my people, I promptly converted the PRL website from Racket-generating-HTML to the fine <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> preprocessor language (commit <a href="https://github.com/nuprl/website/commit/a0600d32fec4bd70c5530b2717aec32979d634f7"><code>a0600d</code></a>) This bold action polarized the community.</p> + +<blockquote> + <p>I can&rsquo;t read the source anymore! Is this really an improvement?</p></blockquote> + +<p>Fear not, citizens. The switch to <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> was the right choice, and you too can learn to read the source code.</p> + +<h2 id="how-to-read-scribblehtml-programs">How to Read <code>scribble/html</code> Programs</h2> + +<h3 id="basics">Basics</h3> + +<p>Scribble is a language for writing Racket documentation. The key innovation in Scribble is the <em>@-expression</em> (read: &ldquo;at expression&rdquo;). The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language combines @-expression syntax with functions that generate HTML.</p> + +<h4 id="-syntax">@-syntax</h4> + +<p><a href="http://www.greghendershott.com/2015/08/at-expressions.html">Greg Hendershott</a> and the <a href="http://docs.racket-lang.org/scribble/reader.html">Scribble Documentation</a> explain @-expressions properly. Here&rsquo;s a short tutorial (Part 1 of 2, &ldquo;the basics&rdquo;):</p> + +<ul> + <li>Scribble programs start in &ldquo;text mode&rdquo;. Every character you type goes straight to the document you are building.</li> + <li>The @-sign toggles to &ldquo;Racket mode&rdquo; for the next expression. In Racket mode, the characters you type will be evaluated as a Racket program to produce part of the document.</li></ul> + +<p><em>Examples:</em> Evaluating <code>"Hello Dave"</code> puts &ldquo;Hello Dave&rdquo; in your document. Evaluating <code>"Hello @Dave"</code> puts &ldquo;Hello ???&rdquo; in your document, where "???" is the value of the variable <code>Dave</code>. Finally if <code>Dave</code> is the name of a function, then <code>"Hello @(Dave)"</code> calls the <code>Dave</code> function with zero arguments and puts whatever it returns into your document.</p> + +<p>To make it easy to interleave text, function calls, and code, Scribble discriminates between 4 kinds of parentheses when they follow an @-sign (Part 2 of 2, &ldquo;the parens&rdquo;):</p> + +<ul> + <li><code>@(f A B)</code> is just like the function call <code>(f A B)</code> in Racket</li> + <li><code>@f[A B]</code> is the same as <code>@(f A B)</code>, but typically more useful because &hellip;</li> + <li><code>@f[A B]{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; to a list of words <code>w*</code>, then calls <code>f</code> just like <code>(apply f A B w*)</code></li> + <li><code>@f{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; and calls <code>f</code> with the results</li> + <li><code>@f|{....}|</code> is similar, but the <code>....</code> are in &ldquo;unescapable text mode&rdquo;</li></ul> + +<p>&ldquo;Unescapable text mode&rdquo; treats @-signs as text instead of toggling between modes.</p> + +<h4 id="generating-html">Generating HTML</h4> + +<p>The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language comes with functions that render HTML. These functions have the same name as the corresponding HTML tag.</p> + +<p>Example program:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this program prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>No surprises.</p> + +<p>One thing that <em>is</em> surprising is how <code>scribble/html</code> handles tag attributes. Every tag-rendering function accepts &ldquo;Racket mode&rdquo; arguments that specify an attribute name and attribute value.</p> + +<p>For example:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">[</span><span class="n">style:</span> <span class="s2">"color:red"</span><span class="p">]{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Hope the output looks familiar. The input syntax is strange, but that&rsquo;s what it is.</p> + +<p>Larger programs print larger webpages. Each page on the PRL website is HTML generated by one <code>scribble/html</code> program.</p> + +<h2 id="why-scribblehtml-is-an-improvement">Why <code>scribble/html</code> is an Improvement</h2> + +<p>Before <code>scribble/html</code>, the PRL website was implemented in <code>scribble/text</code>. A <code>scribble/text</code> program renders and prints text. There is no extra support for HTML.</p> + +<p>To compare, here&rsquo;s the start of the old homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/text</span> +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="s2">"templates.rkt"</span><span class="p">)</span> + +<span class="n">&lt;!DOCTYPE</span> <span class="n">html&gt;</span> +<span class="n">&lt;html</span> <span class="n">lang=</span><span class="s2">"en"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">header</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;body</span> <span class="n">id=</span><span class="s2">"pn-top"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">navbar</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;div</span> <span class="n">class=</span><span class="s2">"jumbotron"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And here is the start of the <code>scribble/html</code>&rsquo;d homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@require</span><span class="p">[</span><span class="s2">"templates.rkt"</span><span class="p">]</span> + +<span class="n">@doctype</span><span class="p">{</span><span class="n">html</span><span class="p">}</span> +<span class="n">@html</span><span class="p">[</span><span class="n">lang:</span> <span class="s2">"en"</span><span class="p">]{</span> + <span class="n">@header</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@body</span><span class="p">[</span><span class="n">id:</span> <span class="s2">"pn-top"</span><span class="p">]{</span> + <span class="n">@navbar</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@div</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"jumbotron"</span><span class="p">]{</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The pages look similar. The new one has more @-signs and parentheses, the old one has more <code>&lt;</code>-signs and quotes. If you were able to edit the old page, you should be able to edit the new page.</p> + +<p>The <strong>key improvement</strong> in the new page is that <strong>common mistakes are now compile-time errors</strong>.</p> + +<ul> + <li> + <p>Before, a typo like <code>&lt;hmtl&gt;</code> would generate an ugly webpage. After, a typo like <code>@hmtl</code> is a syntax error.</p></li> + <li> + <p>Before, a typo like <code>&lt;b&gt;....</code> with no closing tag would generate an ugly webpage. After, a typo like <code>@b{....</code> is a syntax error.</p></li></ul> + +<p>Both flavors of error message come with source-code line numbers. This is very very helpful.</p> + +<h3 id="small-improvements">Small Improvements</h3> + +<h4 id="1-more-functions">1. More Functions</h4> + +<p>Before, the <a href="http://prl.ccs.neu.edu/teaching.html">Teaching page</a> contained some interesting HTML for rendering vertical text (look for the word &ldquo;Semantics&rdquo; to see how this was used):</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"how-to-design-programs"</span><span class="p">&gt;</span>S<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>e<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>m<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>a<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>n<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>t<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>i<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>c<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>s<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>After, the same text is generated from a function call:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@span</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"how-to-design-programs"</span><span class="p">]{</span><span class="n">@vertical-text</span><span class="p">{</span><span class="n">Semantics</span><span class="p">}}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The <code>vertical-text</code> function is simple:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@require</span><span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._only-in))" style="color: inherit">only-in</a></span> <span class="n">racket/list</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span><span class="p">)]</span> + +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">vertical-text</span> <span class="o">.</span> <span class="n">str*</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-~3elist))" style="color: inherit">string-&gt;list</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._append*))" style="color: inherit">append*</a></span> <span class="n">str*</span><span class="p">))</span> <span class="p">(</span><span class="n">br</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="2-more-structure-less-boilerplate">2. More Structure, Less Boilerplate</h4> + +<p>Here&rsquo;s part of the old definition of &ldquo;Ben Greenman&rdquo; on the <a href="http://prl.ccs.neu.edu/people.html">People page</a>:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"row pn-person"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-row-eq-height"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-photo"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"img-wrapper"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">"img/ben_greenman.jpg"</span> <span class="na">title</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="na">alt</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="p">/&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-9"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-4 pn-contact"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"pn-name"</span><span class="p">&gt;</span>Ben Greenman<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + Advisor: Matthias Felleisen<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"mailto:types@"</span><span class="err">@"</span><span class="na">ccs</span><span class="err">.</span><span class="na">neu</span><span class="err">.</span><span class="na">edu</span><span class="err">"</span><span class="p">&gt;</span>types@"@"ccs.neu.edu<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"http://www.ccs.neu.edu/home/types"</span><span class="p">&gt;</span>www.ccs.neu.edu/home/types<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-muted col-md-offset-5"</span><span class="p">&gt;</span> + Joined 2014 + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-bio"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>I like constructions .... <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new definition uses a helper function with keyword arguments for each &ldquo;field&rdquo; of the person:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@person</span><span class="p">[</span><span class="kd">#:name</span> <span class="s2">"Ben Greenman"</span> + <span class="kd">#:title</span> <span class="s2">"Advisor: Matthias Felleisen"</span> + <span class="kd">#:e-mail</span> <span class="s2">"types@ccs.neu.edu"</span> + <span class="kd">#:website</span> <span class="s2">"http://ccs.neu.edu/home/types"</span> + <span class="kd">#:history</span> <span class="n">@list</span><span class="p">[</span><span class="s2">"Joined 2014"</span><span class="p">]</span> + <span class="kd">#:img</span> <span class="s2">"ben_greenman.jpg"</span><span class="p">]{</span> + <span class="n">I</span> <span class="n">like</span> <span class="n">constructions</span> <span class="n">....</span> +<span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="3-less-string-formatting">3. Less String-Formatting</h4> + +<p>Before, the code did a lot of string formatting (<a href="https://github.com/nuprl/website/commit/a0600d#diff-1921e33ce89be28dd277cf1c7880d1beL9">link</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/creatingunits.html#(form._((lib._racket/unit..rkt)._link))" style="color: inherit">link</a></span> <span class="n">url</span> <span class="n">body</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-append))" style="color: inherit">string-append</a></span> <span class="s2">"&lt;a href=</span><span class="se">\"</span><span class="s2">"</span> <span class="n">url</span> <span class="s2">"</span><span class="se">\"</span><span class="s2">&gt;"</span> <span class="n">body</span> <span class="s2">"&lt;/a&gt;"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new code has no need for such helper functions.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@a</span><span class="p">[</span><span class="n">href:</span> <span class="n">url</span> <span class="n">body</span><span class="p">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="bottom-line">Bottom Line</h4> + +<p>Scribble is a good language for making static HTML pages.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://docs.racket-lang.org/pollen/index.html">Pollen</a></li> + <li><a href="https://github.com/vishesh/racketscript">RacketScript</a></li> + <li>Other websites built using <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a>: (1) <a href="http://nanopass.org/">nanopass.github.io</a> (<a href="https://github.com/nanopass/nanopass.github.io">source code</a>), (2) <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> (<a href="https://github.com/nuprl/gtp">source code</a>).</li> + <li><a href="http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/">Notes from a Gradual Typing Across the Spectrum PI meeting</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/System-Administration.atom.xml b/blog/feeds/System-Administration.atom.xml new file mode 100644 index 00000000..7237f690 --- /dev/null +++ b/blog/feeds/System-Administration.atom.xml @@ -0,0 +1,70 @@ + + + PRL Blog: Posts tagged 'System Administration' + + + urn:http-prl-ccs-neu-edu:-blog-tags-System-Administration-html + 2016-10-17T21:48:25Z + + Emacs daemon for fast editor startup + + urn:http-prl-ccs-neu-edu:-blog-2016-10-17-emacs-daemon-for-fast-editor-startup + 2016-10-17T21:48:25Z + 2016-10-17T21:48:25Z + + Gabriel Scherer + +<p>In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs&rsquo; bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors &mdash; on each git commit for example, or when I use a <a href="https://addons.mozilla.org/en-US/firefox/addon/its-all-text/">Firefox extension</a> to edit a textarea content in a proper editor.</p> + +<p>In this blog post, I describe how to use <code>emacsclient</code> to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.</p> +<!-- more--> + +<p>Emacs has long supported a client/server mode: a daemon emacs instance is loaded in the background, and whenever you request a new emacs window you can creates a new frame (~ window) using the same instance. This means that the startup time is dramatically reduced after the daemon is launched, as for example the execution of your personal configuration code does not have to be repeated.</p> + +<p>To use it, I have this code as <code>/usr/bin/editor</code>:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="ch">#!/bin/bash</span> +emacsclient -a <span class="s2">""</span> -c <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The empty <code>-a</code> parameter means that if no daemon exists, it should start one in the background and retry. The <code>-c</code> option means that a new frame (window) should be created instead of reusing an existing one. <code>"$@"</code>means that when the script is invoked with a path as command-line parameter (<code>editor /tmp/foo.txt</code>), the corresponding file will be opened.</p> + +<p>Finally, my <code>.bash_profile</code> sets the <code>EDITOR</code> variable to <code>editor</code> (<code>export EDITOR=/usr/bin/editor</code>); this environment variable is what most tools (git included) will use to invoke a text editor.</p> + +<p>On my machine, starting the daemon takes 1.4s. Creating a client windows takes around 0.150s.</p> + +<p>If you want to control the environment in which the daemon process is started, you can launch it explicitly by running <code>emacs --daemon</code>.</p> + +<p>Cool kids use <a href="http://spacemacs.org/">Spacemacs</a> these days, which comes with all sort of convenient settings built in, and I&rsquo;m told that it does daemonization out of the box. I haven&rsquo;t taken the time to give Spacemacs a try yet.</p> + +<p>Finally, sometimes having all editor windows share the same process is not the right thing, because I do stuff which makes Emacs a bit unstable. (It&rsquo;s not very good at asynchronous communication with the rest of the world, so for example accessing a file through SSH from Emacs can hang the process when network goes bad.). I&rsquo;ve been bitten a few times by a crash of all editor windows at the same time, and since then, when I know I&rsquo;m going to do &ldquo;heavy stuff&rdquo;, I launch a separate process for it (just <code>emacs</code> instead of <code>editor</code> or <code>emacsclient</code>).</p> + +<p>P.S.: To precisely measure the startup time, ask Emacs to evaluate a Lisp expression on startup, to kill it immediately.:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span>$ <span class="nb">time</span> emacs --eval <span class="s2">"(save-buffers-kill-terminal)"</span> +$ <span class="nb">time</span> emacsclient -a <span class="s1">&#39;&#39;</span> -c -e <span class="s2">"(save-buffers-kill-terminal)"</span> +</pre></div> +</td></tr></tbody></table> +</div> \ No newline at end of file diff --git a/blog/feeds/System-Administration.rss.xml b/blog/feeds/System-Administration.rss.xml new file mode 100644 index 00000000..2fa22b14 --- /dev/null +++ b/blog/feeds/System-Administration.rss.xml @@ -0,0 +1,70 @@ + + + + PRL Blog: Posts tagged 'System Administration' + PRL Blog: Posts tagged 'System Administration' + http://prl.ccs.neu.edu/blog/tags/System-Administration.html + Mon, 17 Oct 2016 21:48:25 UT + Mon, 17 Oct 2016 21:48:25 UT + 1800 + + Emacs daemon for fast editor startup + http://prl.ccs.neu.edu/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/?utm_source=System-Administration&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-17-emacs-daemon-for-fast-editor-startup + Mon, 17 Oct 2016 21:48:25 UT + Gabriel Scherer + +<p>In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs&rsquo; bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors &mdash; on each git commit for example, or when I use a <a href="https://addons.mozilla.org/en-US/firefox/addon/its-all-text/">Firefox extension</a> to edit a textarea content in a proper editor.</p> + +<p>In this blog post, I describe how to use <code>emacsclient</code> to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.</p> +<!-- more--> + +<p>Emacs has long supported a client/server mode: a daemon emacs instance is loaded in the background, and whenever you request a new emacs window you can creates a new frame (~ window) using the same instance. This means that the startup time is dramatically reduced after the daemon is launched, as for example the execution of your personal configuration code does not have to be repeated.</p> + +<p>To use it, I have this code as <code>/usr/bin/editor</code>:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="ch">#!/bin/bash</span> +emacsclient -a <span class="s2">""</span> -c <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The empty <code>-a</code> parameter means that if no daemon exists, it should start one in the background and retry. The <code>-c</code> option means that a new frame (window) should be created instead of reusing an existing one. <code>"$@"</code>means that when the script is invoked with a path as command-line parameter (<code>editor /tmp/foo.txt</code>), the corresponding file will be opened.</p> + +<p>Finally, my <code>.bash_profile</code> sets the <code>EDITOR</code> variable to <code>editor</code> (<code>export EDITOR=/usr/bin/editor</code>); this environment variable is what most tools (git included) will use to invoke a text editor.</p> + +<p>On my machine, starting the daemon takes 1.4s. Creating a client windows takes around 0.150s.</p> + +<p>If you want to control the environment in which the daemon process is started, you can launch it explicitly by running <code>emacs --daemon</code>.</p> + +<p>Cool kids use <a href="http://spacemacs.org/">Spacemacs</a> these days, which comes with all sort of convenient settings built in, and I&rsquo;m told that it does daemonization out of the box. I haven&rsquo;t taken the time to give Spacemacs a try yet.</p> + +<p>Finally, sometimes having all editor windows share the same process is not the right thing, because I do stuff which makes Emacs a bit unstable. (It&rsquo;s not very good at asynchronous communication with the rest of the world, so for example accessing a file through SSH from Emacs can hang the process when network goes bad.). I&rsquo;ve been bitten a few times by a crash of all editor windows at the same time, and since then, when I know I&rsquo;m going to do &ldquo;heavy stuff&rdquo;, I launch a separate process for it (just <code>emacs</code> instead of <code>editor</code> or <code>emacsclient</code>).</p> + +<p>P.S.: To precisely measure the startup time, ask Emacs to evaluate a Lisp expression on startup, to kill it immediately.:</p> + +<div class="brush: sh"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span>$ <span class="nb">time</span> emacs --eval <span class="s2">"(save-buffers-kill-terminal)"</span> +$ <span class="nb">time</span> emacsclient -a <span class="s1">&#39;&#39;</span> -c -e <span class="s2">"(save-buffers-kill-terminal)"</span> +</pre></div> +</td></tr></tbody></table> +</div> \ No newline at end of file diff --git a/blog/feeds/Takikawa-constant.atom.xml b/blog/feeds/Takikawa-constant.atom.xml new file mode 100644 index 00000000..650df6cf --- /dev/null +++ b/blog/feeds/Takikawa-constant.atom.xml @@ -0,0 +1,133 @@ + + + PRL Blog: Posts tagged 'Takikawa constant' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Takikawa-constant-html + 2018-05-08T15:37:37Z + + Sampling Gradual Typing Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + 2018-05-08T15:37:37Z + 2018-05-08T15:37:37Z + Ben Greenman + Zeina Migeed + + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> \ No newline at end of file diff --git a/blog/feeds/Takikawa-constant.rss.xml b/blog/feeds/Takikawa-constant.rss.xml new file mode 100644 index 00000000..c83dcdd0 --- /dev/null +++ b/blog/feeds/Takikawa-constant.rss.xml @@ -0,0 +1,131 @@ + + + + PRL Blog: Posts tagged 'Takikawa constant' + PRL Blog: Posts tagged 'Takikawa constant' + http://prl.ccs.neu.edu/blog/tags/Takikawa-constant.html + Tue, 08 May 2018 15:37:37 UT + Tue, 08 May 2018 15:37:37 UT + 1800 + + Sampling Gradual Typing Performance + http://prl.ccs.neu.edu/blog/2018/05/08/sampling-gradual-typing-performance/?utm_source=Takikawa-constant&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + Tue, 08 May 2018 15:37:37 UT + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> \ No newline at end of file diff --git a/blog/feeds/Yoneda.atom.xml b/blog/feeds/Yoneda.atom.xml new file mode 100644 index 00000000..e318e0ed --- /dev/null +++ b/blog/feeds/Yoneda.atom.xml @@ -0,0 +1,164 @@ + + + PRL Blog: Posts tagged 'Yoneda' + + + urn:http-prl-ccs-neu-edu:-blog-tags-Yoneda-html + 2017-08-28T10:30:00Z + + Closure Conversion as CoYoneda + + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + 2017-08-28T10:30:00Z + 2017-08-28T10:30:00Z + + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/Yoneda.rss.xml b/blog/feeds/Yoneda.rss.xml new file mode 100644 index 00000000..c547f42e --- /dev/null +++ b/blog/feeds/Yoneda.rss.xml @@ -0,0 +1,164 @@ + + + + PRL Blog: Posts tagged 'Yoneda' + PRL Blog: Posts tagged 'Yoneda' + http://prl.ccs.neu.edu/blog/tags/Yoneda.html + Mon, 28 Aug 2017 10:30:00 UT + Mon, 28 Aug 2017 10:30:00 UT + 1800 + + Closure Conversion as CoYoneda + http://prl.ccs.neu.edu/blog/2017/08/28/closure-conversion-as-coyoneda/?utm_source=Yoneda&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + Mon, 28 Aug 2017 10:30:00 UT + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/about.atom.xml b/blog/feeds/about.atom.xml new file mode 100644 index 00000000..5689318d --- /dev/null +++ b/blog/feeds/about.atom.xml @@ -0,0 +1,39 @@ + + + PRL Blog: Posts tagged 'about' + + + urn:http-prl-ccs-neu-edu:-blog-tags-about-html + 2016-04-29T14:50:29Z + + Welcome to the PRL blog + + urn:http-prl-ccs-neu-edu:-blog-2016-04-29-welcome-to-the-prl-blog + 2016-04-29T14:50:29Z + 2016-04-29T14:50:29Z + + Ben Greenman + +<p>Greetings, ground rules, hopes, dreams, and notes for contributors. Welcome aboard.</p> +<!-- more--> + +<p>Earlier this year, the Programming Research Lab (PRL) was blessed with a new postdoc: <a href="http://gallium.inria.fr/~scherer/">Gabriel Scherer</a> from INRIA Paris-Rocquencourt, France. Ever since Gabriel arrived things have been changing here in Boston. We now have homemade bread on the first Tuesday of every month, <a href="https://en.wikipedia.org/wiki/Orange_flower_water">orange water</a> crepes after holidays, and someone new to go out for bubble tea with in between. All that and an enthusiastic colleague and researcher.</p> + +<p>In his spare time between lobbying the CS department for an espresso machine and building multi-language compilers, Gabriel is also a champion of open access. Hence this blog, a window into the life and times of PRL students made possible by Gabriel&rsquo;s tactical prodding and careful delegation of responsibilities. Anything you might read about in a rejected conference paper or hear over coffee is fair game here: the goal is to give the wide world a glimpse of our lab and people.</p> + +<h2 id="for-contributors">For Contributors</h2> + +<p>These pages are generated using Greg Hendershott&rsquo;s <a href="https://github.com/greghendershott/frog">frog</a> static website generator. To create a new post:</p> + +<ol> + <li>Clone or fork the <a href="https://github.com/nuprl/nuprl.github.io">nuprl.github.io</a> repository</li> + <li>Check out a new git branch for your post</li> + <li>Run <code>cd blog; raco frog -n "TITLE"</code> to build a template for a new post</li> + <li>Add content to the new markdown file (under <code>_src/posts</code>)</li> + <li>Rebuild the blog with <code>raco frog -b</code></li> + <li>Run <code>cd ..; raco frog -p</code> to start a web server and view your changes at <a href="http://localhost:3000/">http://localhost:3000/</a></li> + <li>Send a pull request to the <a href="https://github.com/nuprl/nuprl.github.io">nuprl.github.io</a> repo</li></ol> + +<p>An open pull request is the best place to ask questions about the formatting or content of a post. We promise that within a few days of opening a PR someone with push access will reply with feedback or merge the request.</p> + +<p>Contributions are open to anyone: current labmates, alumni, friends from the Racket mailing list, and even recovering C programmers. One should have a strong connection to Northeastern or our research, but even that is not strictly necessary. Visitors are always welcome to the PRL.</p> \ No newline at end of file diff --git a/blog/feeds/about.rss.xml b/blog/feeds/about.rss.xml new file mode 100644 index 00000000..bf93a943 --- /dev/null +++ b/blog/feeds/about.rss.xml @@ -0,0 +1,39 @@ + + + + PRL Blog: Posts tagged 'about' + PRL Blog: Posts tagged 'about' + http://prl.ccs.neu.edu/blog/tags/about.html + Fri, 29 Apr 2016 14:50:29 UT + Fri, 29 Apr 2016 14:50:29 UT + 1800 + + Welcome to the PRL blog + http://prl.ccs.neu.edu/blog/2016/04/29/welcome-to-the-prl-blog/?utm_source=about&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-04-29-welcome-to-the-prl-blog + Fri, 29 Apr 2016 14:50:29 UT + Ben Greenman + +<p>Greetings, ground rules, hopes, dreams, and notes for contributors. Welcome aboard.</p> +<!-- more--> + +<p>Earlier this year, the Programming Research Lab (PRL) was blessed with a new postdoc: <a href="http://gallium.inria.fr/~scherer/">Gabriel Scherer</a> from INRIA Paris-Rocquencourt, France. Ever since Gabriel arrived things have been changing here in Boston. We now have homemade bread on the first Tuesday of every month, <a href="https://en.wikipedia.org/wiki/Orange_flower_water">orange water</a> crepes after holidays, and someone new to go out for bubble tea with in between. All that and an enthusiastic colleague and researcher.</p> + +<p>In his spare time between lobbying the CS department for an espresso machine and building multi-language compilers, Gabriel is also a champion of open access. Hence this blog, a window into the life and times of PRL students made possible by Gabriel&rsquo;s tactical prodding and careful delegation of responsibilities. Anything you might read about in a rejected conference paper or hear over coffee is fair game here: the goal is to give the wide world a glimpse of our lab and people.</p> + +<h2 id="for-contributors">For Contributors</h2> + +<p>These pages are generated using Greg Hendershott&rsquo;s <a href="https://github.com/greghendershott/frog">frog</a> static website generator. To create a new post:</p> + +<ol> + <li>Clone or fork the <a href="https://github.com/nuprl/nuprl.github.io">nuprl.github.io</a> repository</li> + <li>Check out a new git branch for your post</li> + <li>Run <code>cd blog; raco frog -n "TITLE"</code> to build a template for a new post</li> + <li>Add content to the new markdown file (under <code>_src/posts</code>)</li> + <li>Rebuild the blog with <code>raco frog -b</code></li> + <li>Run <code>cd ..; raco frog -p</code> to start a web server and view your changes at <a href="http://localhost:3000/">http://localhost:3000/</a></li> + <li>Send a pull request to the <a href="https://github.com/nuprl/nuprl.github.io">nuprl.github.io</a> repo</li></ol> + +<p>An open pull request is the best place to ask questions about the formatting or content of a post. We promise that within a few days of opening a PR someone with push access will reply with feedback or merge the request.</p> + +<p>Contributions are open to anyone: current labmates, alumni, friends from the Racket mailing list, and even recovering C programmers. One should have a strong connection to Northeastern or our research, but even that is not strictly necessary. Visitors are always welcome to the PRL.</p> \ No newline at end of file diff --git a/blog/feeds/adjunction.atom.xml b/blog/feeds/adjunction.atom.xml new file mode 100644 index 00000000..88138951 --- /dev/null +++ b/blog/feeds/adjunction.atom.xml @@ -0,0 +1,85 @@ + + + PRL Blog: Posts tagged 'adjunction' + + + urn:http-prl-ccs-neu-edu:-blog-tags-adjunction-html + 2016-11-16T00:00:00Z + + Understanding Constructive Galois Connections + + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + 2016-11-16T00:00:00Z + 2016-11-16T00:00:00Z + + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/adjunction.rss.xml b/blog/feeds/adjunction.rss.xml new file mode 100644 index 00000000..8eca79fd --- /dev/null +++ b/blog/feeds/adjunction.rss.xml @@ -0,0 +1,85 @@ + + + + PRL Blog: Posts tagged 'adjunction' + PRL Blog: Posts tagged 'adjunction' + http://prl.ccs.neu.edu/blog/tags/adjunction.html + Wed, 16 Nov 2016 00:00:00 UT + Wed, 16 Nov 2016 00:00:00 UT + 1800 + + Understanding Constructive Galois Connections + http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/?utm_source=adjunction&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + Wed, 16 Nov 2016 00:00:00 UT + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/all.atom.xml b/blog/feeds/all.atom.xml new file mode 100644 index 00000000..22e03edd --- /dev/null +++ b/blog/feeds/all.atom.xml @@ -0,0 +1,3645 @@ + + + PRL Blog: PRL Blog + + + urn:http-prl-ccs-neu-edu:-blog-index-html + 2022-02-16T00:00:42Z + + [Ř Overview I (cross-post)](https://www.o1o.ch/lab/entry/1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc.html) + + urn:http-prl-ccs-neu-edu:-blog-2022-02-16-r-CC-8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html + 2022-02-16T00:00:42Z + 2022-02-16T00:00:42Z + + Olivier Flückiger + + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + 2022-01-06T17:56:08Z + 2022-01-06T17:56:08Z + + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> + + Deep and Shallow Types + + urn:http-prl-ccs-neu-edu:-blog-2020-12-23-deep-and-shallow-types + 2020-12-23T18:21:55Z + 2020-12-23T18:21:55Z + + Ben Greenman + +<p>I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:</p> + +<ul> + <li><a href="http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020</a></li></ul> + +<p>To the PRL: thanks for a wonderful 6.5 years.</p> +<!-- more--> + +<h3 id="abstract">Abstract</h3> + +<blockquote> + <p>The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound&rsquo;&rsquo; mixed-typed languages from others that enforce type obligations locally rather than globally.</p> + <p>In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.</p> + <p>My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.</p> + <p>I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.</p></blockquote> + +<p>Next year, I&rsquo;ll be a <a href="https://cifellows2020.org">CI Fellow</a> at Brown.</p> + + Transient for Optional and Keyword Functions + + urn:http-prl-ccs-neu-edu:-blog-2020-11-12-transient-for-optional-and-keyword-functions + 2020-11-12T10:15:16Z + 2020-11-12T10:15:16Z + + Ben Greenman + +<p>A short adventure into the depths of optional and/or keyword functions in Racket.</p> +<!-- more--> + +<hr /> + +<p>Transient, or rather <em>the Transient semantics for a mixed-typed language</em>, is one way to let statically-typed code safely interact with untyped code. You can read all about it in <a href="http://hdl.handle.net/2022/23172">Michael Vitousek&rsquo;s 2019 dissertation</a> or <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my 2020 dissertation</a>, and you can see how it compares to other mixed-typed semantics <a href="http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">here</a>. The idea is to give up on <a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">behavioral type guarantees</a> and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called <em>shape checks</em>; for example:</p> + +<ul> + <li>if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;</li> + <li>if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and</li> + <li>if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.</li></ul> + +<p>Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type <strong>T</strong> into a shape check, and we have a function with type <strong>T</strong> that needs to check its inputs. The question is how to actually do the check in Racket v7.9.</p> + +<p>In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if <strong>T = (-&gt; Symbol Boolean)</strong> then we need to check the shape <strong>symbol?</strong> of the domain type <strong>Symbol</strong>:</p> + +<pre><code>;; source code +(: f (-&gt; Symbol Boolean)) +(define (f sym) + (eq? sym 'hola)) + +;; ===&gt; + +;; imaginary (but realistic) rewritten code +(define (f sym) + (assert sym symbol?) + (eq? sym 'hola))</code></pre> + +<p>A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type <strong>T</strong> accepts 1 to 3 inputs:</p> + +<pre><code>;; source code +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) +(define (g #:a a [b 'b] #:c [c #f]) + (if a b (if c 'left 'right))) + +;; ===&gt; + +;; imaginary, unrealistic rewritten code +(define (g #:a a [b 'b] #:c [c #f]) + (assert a boolean?) + (assert b symbol?) + (assert c void?) + (if a b (if c 'left 'right)))</code></pre> + +<p>Good &mdash; we basically know what we want. If the Racket core language had optional and keyword functions, then we&rsquo;d be done.</p> + +<p>But no, Racket expands these optional/keyword functions into primitive <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._lam))"><strong>lambda</strong></a> and <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._case-lam))"><strong>case-lambda</strong></a> forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.</p> + +<p>Let&rsquo;s keep digging.</p> + +<p>From now on, &ldquo;Shallow&rdquo; or &ldquo;Shallow TR&rdquo; refers to my implementation of Transient for Typed Racket (TR). We&rsquo;ll talk about Shallow instead of &ldquo;Transient&rdquo; in case future work reveals a better way to implement the Transient idea.</p> + +<h2 id="false-start-follow-the-type">False Start: Follow the Type</h2> + +<p>Beware &mdash; Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function <strong>g</strong> above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional <strong>#:c</strong> keyword lead to a shape-check failure because the variable <strong>c</strong> gets the default value <strong>#f</strong> instead of a void value. Concretely, the third assert from above fails:</p> + +<pre><code>(define (g #:a a [b 'b] #:c [c #f]) + .... + (assert c void?) ;; fails if c is the #f default value + ....)</code></pre> + +<p>The problem arises from subtyping. According to the annotations, the function <strong>g</strong> has an external type that is less precise than the internal type that validates the function body:</p> + +<pre><code>;; external type T +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) + +;; internal type T2, subtype of external (T2 &lt;: T), validates body +(: g (-&gt;* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))</code></pre> + +<p>Thanks to this external / internal distinction, the following easy rewrite idea, <em>Solution 0</em>, fails. Despite the failure, this first solution is a useful starting point for a success.</p> + +<h4 id="solution-0-step-1-mimic-the-typechecker">Solution 0, Step 1: Mimic the Typechecker</h4> + +<p>Shallow TR uses the same type checker as classic <em>Deep</em> TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.</p> + +<p>Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.</p> + +<p>Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (<a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-expr-unit.rkt#L274-L295">typecheck/tc-expr-unit.rkt</a>). The omitted code (<strong>&hellip;.</strong>) does actual type checking:</p> + +<pre><code>(define (tc-expr/check/internal form expected-type) + .... + (syntax-parse form + #:literal-sets (kernel-literals tc-expr-literals) + .... + [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^) + ....] + [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>Ok! Those two patterns say a lot about the expansion of optional and keyword functions:</p> + +<ol> + <li>Both forms expand to a <strong>let-values</strong> that binds one function <strong>fun</strong>.</li> + <li>TR uses the syntax classes <strong>kw-lambda^</strong> and <strong>opt-lambda^</strong> to tell these particular <strong>let-values</strong> apart from others.</li></ol> + +<p>Shallow TR can use exactly these patterns to recognize optional/keyword functions.</p> + +<h4 id="solution-0-step-2-parse-the-domain-type">Solution 0, Step 2: Parse the Domain Type</h4> + +<p>Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function&rsquo;s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:</p> + +<pre><code> [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/type-table.rkt#L80"><strong>type-of</strong></a> function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, <strong>fun</strong>, has a type. Great.</p> + +<p>Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:</p> + +<ul> + <li>The type of <strong>fun</strong> could be a straightforward <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L693"><strong>Fun type</strong></a>, but it could also be a: <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L701"><strong>DepFun type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L521"><strong>Poly type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L531"><strong>PolyDots type</strong></a>, or even a <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L900"><strong>Union type</strong></a>.</li> + <li>Each part of the domain type corresponds to one parameter of the <strong>fun</strong> expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in <strong>fun</strong>, or the mandatory keywords?</li></ul> + +<h4 id="solution-0-step-3-insert-a-shape-check">Solution 0, Step 3: Insert a Shape Check</h4> + +<p>Once we have the target <strong>fun</strong> expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse <strong>fun</strong> to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting <strong>fun</strong> goes something like this:</p> + +<pre><code>(syntax-parse fun + [(#%plain-lambda formals . body) + #:with (shape-check ...) + (make-shape-checks #'formals (type-of fun)) + #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])</code></pre> + +<p>The rewritten function executes shape checks immediately, and then proceeds with the <strong>body</strong> after validating each actual parameter.</p> + +<h2 id="on-the-trail-optkey-expansion">On the Trail: optkey Expansion</h2> + +<p>Our <em>Solution 0</em> fails because the type of the <strong>fun</strong> expression that it gets from the type-checked code is an external type. In terms of the <strong>g</strong> function from above, <em>Solution 0</em> uses the type <strong>Void</strong> instead of the internal type <strong>(U Void #f)</strong> to check the <strong>c</strong> parameter. To get internal types, we need to look closer at <strong>fun</strong> and the rest of the optional/keyword expansion.</p> + +<p>Let&rsquo;s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.</p> + +<p>If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function <strong>X</strong> create a module <strong>test.rkt</strong> like this:</p> + +<pre><code>#lang racket/base + +(define _ignore + (let () + X + (void)))</code></pre> + +<p>Invoke the expander with <code>raco expand test.rkt &gt; test.rkt.txt</code> and explore the generated <strong>.txt</strong> file.</p> + +<h3 id="example-1-mandatory-keyword">Example 1: mandatory keyword</h3> + +<p>The source is a function with one mandatory positional argument and one optional positional argument.</p> + +<pre><code>(lambda (x [y 0]) + (+ x y))</code></pre> + +<p>Expansion generates a <strong>case-lambda</strong> that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function <strong>F</strong> that expects two arguments, resolves defaults in a different way, and executes the function body.</p> + +<pre><code>(let-values (((F) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F x '0)) + ((x y1) (#%app F x y1))))</code></pre> + +<p>Note: the expression <strong>(if &rsquo;#f &rsquo;0 y1)</strong> in the generated <strong>F</strong> function is equal to <strong>y1</strong> alone. In general, the <strong>if</strong> is for default expressions. (<a href="https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/">Unlike Python</a>, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a <strong>#f</strong> test. A general-purpose optimizer can remove this test before the code runs.</p> + +<h3 id="example-2">Example 2:</h3> + +<p>The source is a function with one mandatory positional argument and one mandatory keyword argument:</p> + +<pre><code>(lambda (x #:y y) + (+ x y))</code></pre> + +<p>Expansion generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> expects a list of keywords, a list of arguments, and a final argument. The purpose of <strong>F1</strong> is to organize a call to <strong>F0</strong>.</li> + <li><strong>lifted/2</strong> is the constructor for a generated struct type. Other functions help the struct call <strong>F1</strong>. Nevermind the details; I don&rsquo;t fully understand them either.</li></ul> + +<p>The important piece for Shallow TR is the <strong>F0</strong> function because the goal of rewriting is to protect the original function body against untyped inputs.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) y1)) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y1) (#%app car given-args))) + (#%app F0 y1 x3))))) + (#%app + lifted/2 + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l2571) given-kws)) + (if (#%app pair? l2571) + (if (#%app eq? (#%app car l2571) '#:y) + (#%app null? (#%app cdr l2571)) + '#f) + '#f)) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + '(#:y) + '(#:y))))</code></pre> + +<h3 id="example-3">Example 3:</h3> + +<p>The source is a function with one mandatory positional argument and one optional keyword argument:</p> + +<pre><code>(lambda (x #:y [y 0]) + (+ x y))</code></pre> + +<p>Expansion again generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments, resolves the optional default, and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> calls <strong>F0</strong></li> + <li>At the bottom, there are two <strong>case-lambda</strong> functions that call <strong>F1</strong></li></ul> + +<p>Again, the <strong>F0</strong> function is the focal point for Shallow TR rewriting.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y2) (#%app pair? given-kws))) + (let-values (((y1) + (if y2 (#%app car given-args) '0))) + (#%app F0 y1 x3)))))) + (#%app + make-optional-keyword-procedure + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l1571) given-kws)) + (let-values (((l1571) + (if (#%app null? l1571) + l1571 + (if (#%app eq? (#%app car l1571) '#:y) + (#%app cdr l1571) + l1571)))) + (#%app null? l1571))) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + null + '(#:y) + (case-lambda + ((x) (#%app F1 null null x))))))</code></pre> + +<h2 id="solution-the-shallow-tr-rewrite-strategy">Solution: The Shallow TR Rewrite Strategy</h2> + +<p>All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a <strong>let-values</strong> form:</p> + +<pre><code>(let-values (((f) fun)) . body)</code></pre> + +<p>Furthermore, the generated <strong>fun</strong> is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the <strong>fun</strong> from <em>Example 3</em> again; it has formal parameters for the keyword arg. and the mandatory arg., and one <strong>let-values</strong> to resolve each parameter:</p> + +<pre><code> (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y)))))</code></pre> + +<p>Another experiment with <strong>type-of</strong> shows that the right-hand side of each <strong>let-values</strong> has an internal type annotation. Excellent! Both <strong>(type-of x3)</strong> and <strong>(type-of (if &rsquo;#f &rsquo;0 y1))</strong> are the right types for shape checks. Shallow TR can:</p> + +<ul> + <li>inspect the <strong>let-values</strong> one-by-one;</li> + <li>convert the type of each right-hand expression to a shape predicate; and</li> + <li>rewrite each right-hand <strong>expr</strong> into <strong>(assert expr shape?)</strong>.</li></ul> + +<p>This should work! In fact, we can do slightly better:</p> + +<ul> + <li>when the right-hand expression is a conditional <strong>(if test default-expr supplied-arg)</strong></li> + <li>then Shallow only needs to check the supplied arg: <strong>(if test default-expr (assert supplied-arg shape?))</strong></li></ul> + +<p>Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.</p> + +<h2 id="a-problem-with-methods-and-a-bugfix">A Problem with Methods and a Bugfix</h2> + +<p>Currently, Shallow TR rewrites optional and keyword functions using the <strong>let-values</strong> plan described above. Each formal parameter has one <strong>let-values</strong> binding, and the type on each bound expression defines the shape check.</p> + +<p>Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we <a href="https://github.com/racket/racket/pull/3182">fixed the class expander</a>.</p> + +<p>First, here is a class with one method that runs correctly. The method <strong>f</strong> accepts an optional positional argument <strong>x</strong>; the default value of <strong>x</strong> is the current value of the field <strong>my-num</strong> (fields are mutable):</p> + +<pre><code>(define c0% + (class object% + (super-new) + (field (my-num 2)) + (define/public (f [x my-num]) + (+ x x))))</code></pre> + +<p>Second, here is a similar method that fails. This time, the default is an immediate value <strong>2</strong>:</p> + +<pre><code>(define c1% + (class object% + (super-new) + (define/public (f [x 2]) + (+ x x))))</code></pre> + +<p>Running a call <strong>(send o1 f)</strong> used to raise a shape-check failure about a strange value:</p> + +<blockquote> + <p>shape error: Expected a real number, got <code>#&lt;unsafe-undefined&gt;</code></p></blockquote> + +<p>What is going on?</p> + +<p>It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:</p> + +<pre><code>(lambda (x [y z]) + (+ x y))</code></pre> + +<p>Expansion generates a function <strong>F0</strong> that checks for the undefined value, and an outer <strong>case-lambda</strong> that supplies undefined when the default is needed:</p> + +<pre><code>(let-values (((F0) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) + (if (#%app eq? y1 unsafe-undefined) + z + y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F0 x unsafe-undefined)) + ((x y1) (#%app F0 x y1))))</code></pre> + +<p>That&rsquo;s the normal way that <strong>unsafe-undefined</strong> shows up: the <a href="https://github.com/racket/racket/blob/c0ff11e27bd28e070c20b7a9b0f7365f8f2b665a/racket/collects/racket/private/kw.rkt">expander for optional/keyword functions</a> looks for default expressions vs. default values and uses the undefined value for expressions.</p> + +<p>Three other facts conspired to make the problem with optional methods:</p> + +<ol> + <li>Typed Racket also looks for default expressions vs. default values (search for <strong>immediate-default</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/base-env/annotate-classes.rkt">here</a>). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the <strong>unsafe-undefined</strong> value (search for <strong>-Unsafe-Undefined</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/kw-types.rkt">here (kw)</a> and <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-lambda-unit.rkt">here (opt)</a>).</li> + <li>The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.</li> + <li>Shallow TR pushes default expression checks <strong>(if test default-expr supplied-arg)</strong> to the <strong>supplied-arg</strong> instead of wrapping the whole <strong>if</strong> form.</li></ol> + +<p>In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound&mdash;but harmless because the false assumption was guarded by an <strong>if</strong> test for <strong>unsafe-undefined</strong>. Running Shallow TR revealed the unsoundness with its eager shape check.</p> + +<p>Again, the resolution was to fix the class expander (<a href="https://github.com/racket/racket/pull/3182">racket/racket #3182</a>). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.</p> + +<h2 id="lessons">Lessons</h2> + +<ol> + <li>Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.</li> + <li>Digging into the expansion is sometimes necessary. There are at least three places that do so&mdash;the class expander, TR, and Shallow TR&mdash;and unfortunately they all need to cooperate.</li> + <li>The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my dissertation</a> lists them all.</li></ol> + + Transient Answers Old Questions + + urn:http-prl-ccs-neu-edu:-blog-2020-10-15-transient-answers-old-questions + 2020-10-15T13:32:12Z + 2020-10-15T13:32:12Z + + Ben Greenman + +<p>Several old questions from the Typed Racket mailing list have new and simple answers under a &ldquo;transient&rdquo; Typed Racket.</p> +<!-- more--> + +<hr /> + +<p>For the past few months, I&rsquo;ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the <a href="https://github.com/racket/typed-racket/pull/952">RFC</a> and <a href="https://github.com/racket/typed-racket/pull/948">pull request</a>.</p> + +<p>The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that&rsquo;s about it. By contrast, types are much stronger in classic Typed Racket.</p> + +<p>Shallow Racket&rsquo;s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the <a href="https://groups.google.com/g/racket-users">Racket mailing list</a>. There were a surprising number of questions that went like this:</p> + +<blockquote> + <p><strong>Q.</strong> Hey, I ran a program expecting <em>X</em> to happen, but <em>Y</em> happened instead. Is this a bug?</p> + <p><strong>A.</strong> No, Typed Racket has to do <em>Y</em> because of its strong types.</p></blockquote> + +<p>&hellip; but changing to shallow types gives the <em>X</em> behavior! Here are their stories.</p> + +<p>Going forward, <strong>Deep</strong> refers to normal Typed Racket and <strong>Shallow</strong> refers to Shallow Typed Racket.</p> + +<hr /> + +<h2 id="higher-order-value-as-any">Higher-Order Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ">groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ</a></p> + +<h4 id="on-20180416-mailoo-wrote">On 2018&ndash;04&ndash;16, <em>mailoo</em> wrote:</h4> + +<blockquote> + <p> I play a little with the &ldquo;Any&rdquo; type (due to &lsquo;dynamic-require&rsquo; which return Any), and I&rsquo;m not able to cast them back in a function.</p> + <p> I (over) simplify my question with this little program :</p></blockquote> + +<pre><code>(: p Any) +(define (p i) (displayln i)) + +; Here I want to get back my function +(define proc (cast p (-&gt; Integer Void))) +(proc 2) </code></pre> + +<blockquote> + <p> but I get this error when I try to execute the function :</p></blockquote> + +<pre><code>; contract violation +; Attempted to use a higher-order value passed as `Any` in untyped code: #&lt;procedure:p&gt; </code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> raises an error because it must enforce the <code>Any</code> type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> prints 2 and returns void. No error. Same goes for dynamic-require.</p> + +<hr /> + +<h2 id="parametric-contract-affects-untyped-code">Parametric Contract Affects Untyped Code</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ">groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ</a></p> + +<h4 id="on-20191215-john-clements-wrote">On 2019&ndash;12&ndash;15, John Clements wrote:</h4> + +<blockquote> + <p> It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:</p></blockquote> + +<pre><code> #lang typed/racket + + (require/typed racket/list + [index-of (All (T) ((Listof T) T -&gt; (U False Natural)))]) + + (index-of '(n s e w) 'n) ;; returns... #f? </code></pre> + +<blockquote> + <p> In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).</p> + <p> I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces parametricity for <code>All</code> types, and this throws off the equality function that index-of uses internally.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> returns 0.</p> + +<p>ps John, thanks very much for working on <a href="https://adventofcode.com">Advent of Code</a> and mailing the list!</p> + +<hr /> + +<h2 id="unable-to-protect-opaque-value-as-any">Unable to Protect Opaque Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ">groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ</a></p> + +<h4 id="on-20191211-marc-kaufmann-wrote">On 2019&ndash;12&ndash;11, Marc Kaufmann wrote:</h4> + +<blockquote> + <p>I have one file called <code>type-test.rkt</code> with the following</p></blockquote> + +<pre><code>#lang typed/racket + +(require (only-in typed/web-server/http response/xexpr response)) + +(provide f2) + +(: f2 (-&gt; (U response Any))) +(define (f2) + (define x '(body (h1 "Try it"))) + (: resp response) + (define resp (response/xexpr x)) + resp)</code></pre> + +<blockquote> + <p>Then I have another <em>untyped</em> file for a servlet:</p></blockquote> + +<pre><code>#lang racket + +(require "type-test.rkt" + web-server/servlet + web-server/servlet-env) + +(define (start req) + (f2)) + +(serve/servlet start + #:servlet-regexp #rx"" + #:launch-browser? #false + #:port 8080)</code></pre> + +<blockquote> + <p>Notice that I am telling [f2] that <code>resp</code> is of type <code>response</code>. Yet, when I run the server with <code>start</code> [&hellip;.] I get the following result:</p> + <p>(f2): Error, see below.</p> + <p>The error is:</p></blockquote> + +<pre><code>f2: broke its own contract + any-wrap/c: Unable to protect opaque value passed as `Any` + value: #&lt;response&gt; + in: the range of + (-&gt; Any)</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the <code>Any</code> type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.</p> + +<hr /> + +<h2 id="type-inference-installs-a-precise-type">Type Inference Installs a Precise Type</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ">groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ</a></p> + +<h4 id="on-20200214-john-clements-wrote">On 2020&ndash;02&ndash;14, John Clements wrote:</h4> + +<blockquote> + <p>I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.</p> + <p>Here’s a program:</p></blockquote> + +<pre><code>#lang typed/racket + +(define-type Store (Mutable-HashTable Integer Value)) +(define-type Value (U Real Boolean String)) + +(define top-store + (cast + (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f))) + Store)) + +(hash-set! top-store 5 1234)</code></pre> + +<blockquote> + <p>It fails with this error:</p></blockquote> + +<pre><code>contract violation +expected: (or/c (and/c byte? positive?) #t #f) +given: 1234 +in: the values of +the 3rd conjunct of +(and/c hash? + hash-mutable? + (hash/c exact-integer? + (or/c (and/c byte? positive?) #t #f) + #:immutable #f))</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p>First off, <strong>Deep</strong> runs fine after swapping <code>cast</code> for <code>ann</code>.</p> + +<p>Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.</p> + +<p>The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.</p> + +<hr /> + +<h2 id="same-arity-functions-in-a-case-lambda">Same-Arity Functions in a Case Lambda</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ">groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ</a></p> + +<h4 id="on-20190705-ryan-kramer-wrote">On 2019&ndash;07&ndash;05, Ryan Kramer wrote:</h4> + +<blockquote> + <p>In the code below, can <code>maybe-car</code> have the given type [&hellip;.]?</p></blockquote> + +<pre><code>#lang typed/racket + +(module untyped racket + (provide maybe-car) + (define (maybe-car x) + (cond + [(pair? x) (car x)] + [else x]))) + +(require/typed + 'untyped + [maybe-car (All (a b) (case-&gt; + (-&gt; (Pairof a b) a) + (-&gt; a a)))])</code></pre> + +<blockquote> + <p>[Current error:]</p></blockquote> + +<pre><code>Type Checker: + Type (All (a b) (case-&gt; (-&gt; (Pairof a b) a) (-&gt; a a))) + could not be converted to a contract: + function type has two cases of arity 1</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the type with a Racket <code>or/c</code> contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don&rsquo;t know which to apply to an incoming function.</p> + +<p>Note, the &ldquo;Type Checker&rdquo; error message is much better than what or/c would give on its own.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> simply checks that maybe-car accepts both arities inside the case-&gt; type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.</p> + +<hr /> + +<h3 id="immutable-type-affects-untyped-code">Immutable Type Affects Untyped Code</h3> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ">groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ</a></p> + +<h4 id="on-20200217-bertrand-augereau-wrote">On 2020&ndash;02&ndash;17, Bertrand Augereau wrote:</h4> + +<blockquote> + <p>Hello everybody, I&rsquo;m trying to gradually type my script to make it a proper app (yes I&rsquo;m a static-ish guy) and I have an issue (Racket 7.6 CS).</p></blockquote> + +<pre><code>; racket_mod.rkt: +#lang racket + +(provide (struct-out s)) +(provide list-of-s) +(provide set-list-of-s!) + +(struct s (a)) +(define list-of-s '()) +(define (set-list-of-s! los) + (set! list-of-s los))</code></pre> + +<pre><code>; racket_mod_typed.rkt: +#lang typed/racket + +(provide (struct-out s2)) +(provide list-of-s2) +(provide set-list-of-s2!) + +(struct s2 ([a : Natural])) +(define list-of-s2 '()) +(define (set-list-of-s2! [los : (Listof s2)]) + (set! list-of-s2 los))</code></pre> + +<pre><code>; racket_main.rkt: +#lang racket + +(require "racket_mod.rkt") +(require "racket_mod_typed.rkt") + +(define los (list (s 1) (s 2))) +(set-list-of-s! los) +(displayln list-of-s) + +(define los2 (list (s2 1) (s2 2))) +(set-list-of-s2! los2) +(displayln list-of-s2)</code></pre> + +<blockquote> + <p>list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces the type of <code>list-of-s2</code> with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.</p> + +<p>Here&rsquo;s a step-by-step breakdown:</p> + +<ol> + <li>the typed module creates an empty list-of-s2</li> + <li>the main module imports the list and receives a new copy</li> + <li>the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable</li> + <li>the main module reads from its copy &mdash; and it&rsquo;s still empty</li></ol> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> lets the original list travel to untyped code. There are no contracts in the way.</p> + +<h2 id="discussion">Discussion</h2> + +<p>Wow! It&rsquo;s great to see that Shallow Racket works &ldquo;as expected&rdquo; on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.</p> + +<p>If you have a similar experience with a deep-types error, let me know.</p> + +<p>Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error &mdash; and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.</p> + +<p>Shallow Typed Racket is coming soon. Follow the <a href="https://github.com/racket/typed-racket/pull/948">pull request</a> or watch the Racket release notes for news.</p> + +<h3 id="links">Links</h3> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">Larger example</a> where Shallow misses an error that Deep catches</li> + <li>Michael M. Vitousek <a href="http://hdl.handle.net/2022/23172">invented</a> the Transient semantics and implemented it in <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>.</li> + <li>My <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-thesis-2020">upcoming dissertation</a> has lots more to say about Shallow Typed Racket.</li></ul> + +<p><em>Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.</em></p> + + The Typed Racket Optimizer vs. Transient + + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + 2020-01-15T12:16:35Z + 2020-01-15T12:16:35Z + + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> + + PRL Offsite 2019 Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2019-12-12-prl-offsite-2019-retrospective + 2019-12-12T12:51:53Z + 2019-12-12T12:51:53Z + Ben Greenman + Olek Gierczak + + Ben Greenman, Olek Gierczak + +<p>On November 11th 2019, the PRL had a private offsite meeting at the <a href="https://mtwyouth.org">More Than Words bookstore</a> in downtown Boston. For future offsite organizers, this post records what happened and how.</p> +<!-- more--> + +<h2 id="early-planning-goals">Early Planning, Goals</h2> + +<p>Every Fall, the PRL holds a kickoff meeting to assign <a href="http://prl.ccs.neu.edu/contact">roles</a> for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.</p> + +<blockquote> + <p>Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the <a href="http://www.warrencenter.com">Warren Conference Center</a> is <a href="https://www.northeastern.edu/events/northeastern-owned-off-campus-venues">Northeastern University&rsquo;s only franchised</a> off-campus venue.</p></blockquote> + +<p>The <strong>motivation</strong> for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students &mdash; both to get to know the group, and to become known. They&rsquo;re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">Spring 2017 edition</a> of HOPL.</p> + +<p>Our primary <strong>goal</strong> this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.</p> + +<p>In light of the main goal, we prefer to call the event an &ldquo;offsite&rdquo; (or &ldquo;offsite meeting&rdquo;) rather than a &ldquo;retreat&rdquo; because the target was an informative day rather than a relaxing one.</p> + +<h2 id="booking-and-logistics">Booking and Logistics</h2> + +<p>Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that&rsquo;s all we had.</p> + +<h3 id="date--nov-11">Date = Nov 11</h3> + +<p>In the beginning, we chose Veterans&rsquo; Day (2019&ndash;11&ndash;11) and Northeastern <a href="https://registrar.northeastern.edu/app/uploads/2019-2020-University-Wide-Calendar-List.pdf">reading day</a> (2019&ndash;12&ndash;05) as possible dates. We ended up with Veterans&rsquo; Day.</p> + +<p>A small number of lab members were opposed to Veterans&rsquo; Day. They gave two reasons: the Fall semester is especially busy, and Veterans&rsquo; Day is a federal holiday.</p> + +<h3 id="venue--mtw">Venue = MTW</h3> + +<p>The first venue we tried was the <a href="http://www.warrencenter.com">Warren Conference Center</a> in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the <a href="https://www.khoury.northeastern.edu/people/chelsea-smith/">Khoury Events Team</a> to reach out on our behalf. They returned with an email address and <a href="/blog/static/warren-center-meetings-and-retreats.pdf">event menu</a> (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.</p> +<!-- Donald Pepple (Northeastern) made contact with Christine Barisano (Framingham)--> + +<p>Second, we contacted the <a href="https://alumni.northeastern.edu/about/alumni-center/">Northeastern University Alumni Center</a>. They were not open on Veterans&rsquo; Day 2019.</p> + +<p>Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: <a href="https://mtwyouth.org">More Than Words</a>.</p> + +<p>More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!</p> + +<blockquote> + <p>More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you&rsquo;ll see a mix of &ldquo;employees&rdquo; and &ldquo;volunteers&rdquo; helping out.</p></blockquote> + +<p>Booking was complicated, though, by the fact that Northeastern requires <a href="http://catalog.northeastern.edu/graduate/health-sciences/academic-policies-procedures/liability-insurance/">liability insurance</a> for all off-campus events. If <strong>you</strong> are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.</p> + +<p>Here are more facts about MTW:</p> + +<ul> + <li>the theater space seats 30 with lots of extra space</li> + <li>the two long tables in the reading room can seat about 20</li> + <li>the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor</li> + <li>there&rsquo;s no whiteboard, but MTW can supply an easel stand</li> + <li>much of the furniture in store is antique and for sale, so be careful using it &mdash; both to avoid damaging it, and because antiques can be oddly-shaped</li> + <li>the bookstore was closed on Veterans&rsquo; Day, but we were able to have our event and buy books</li> + <li>MTW may have fridge space to temporarily hold leftovers</li> + <li>closest T station: <a href="https://www.mbta.com/stops/place-tumnl">Tufts Medical Center</a></li></ul> + +<h3 id="keynote--none">Keynote = None</h3> + +<p>The original, ambitious plan was to invite two keynote speakers &mdash; one working in industry and one from academia &mdash; to enrich the offsite with new knowledge. And because this was the PRL&rsquo;s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)</p> + +<p>We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.</p> + +<p>Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).</p> +<!-- to future planners: we invited Kathi Fisler; she had grant-writing plans for that day and would be interested in coming to a future offsite--> + +<h3 id="food-coffee-tea">Food, Coffee, Tea</h3> + +<p><a href="https://flourbakery.com/">Flour Bakery + Cafe</a> provided lunch. For most days, you can order from Flour using an <a href="https://flourbakery.com/menu/catering/catering-information/">online form</a>. For holidays, you may need to send an email &mdash; that&rsquo;s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan &amp; gluten-free <em>everything spiced</em> salads.</p> + +<p><a href="https://www.trycuppacoffee.com/location/">Cuppacoffee</a> provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything &mdash; the order + napkins, milk, cream cheese, butter, and knives &mdash; ourselves because Cuppacoffee is very close to MTW.</p> + +<p><a href="http://www.cmartboston.com/">CMart</a> sold us water and juices. (There is also a Whole Foods near MTW.)</p> + +<p>At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour&rsquo;s <a href="https://youtu.be/kIbhckqanHI">sticky buns</a>.</p> + +<h3 id="event-program">Event Program</h3> + +<p>The following assumptions/goals constrained the schedule for the day:</p> + +<ul> + <li>give all 17 on-campus Ph.D. students a talk slot; talks must <strong>either</strong> communicate one technical idea from recent work <strong>or</strong> say what led the speaker to apply to a PL Ph.D. program</li> + <li>allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)</li> + <li>do not make the audience sit for more than 1 hour at a time</li> + <li>maximize the number and length of breaks (to encourage discussions)</li> + <li>include a relevant and fun welcome activity</li> + <li>save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions</li></ul> + +<p>We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.</p> + +<p>The activity was <em>codewalks</em>. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition/blob/master/src/main/java/com/seriouscompany/business/java/fizzbuzz/packagenamingpackage/impl/math/arithmetics/IntegerDivider.java">IntegerDivider</a> class from a silly FizzBuzz implementation and a solution to the <a href="https://en.wikipedia.org/wiki/Dutch_national_flag_problem">Dutch national flag problem</a>). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and <a href="https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/">egoless programming</a>.</p> + +<p>The talks ran conference-style. One organizer played &ldquo;session chair&rdquo; to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.</p> + +<p>For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.</p> + +<h2 id="looking-back-at-the-details">Looking Back at the Details</h2> + +<p>Reading Day in the Spring may be a good, free date for future retreats.</p> + +<p>We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.</p> + +<p>The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we&rsquo;d keep the morning breaks short &mdash; just enough to use the restroom and grab a coffee &mdash; and aim for 30-min breaks in the afternoon.</p> + +<p>Groups of three 15-minute talks worked well, but groups of four talks might be equally good.</p> + +<p>Perhaps each speaker should get the chance to pick a talk length. <a href="https://nepls.org/">NEPLS</a>, for example, allows a choice between 5-min. and 30-min. talks.</p> + +<p>The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick &ldquo;topics&rdquo; beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.</p> +<!-- to future organizers: the panel began asking faculty for a mistake in their career and wound up with a defensive flavor--> + +<h2 id="looking-back-overall">Looking Back, Overall</h2> + +<p>This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.</p> + +<p>There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time &mdash; keep the same date &amp; program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.</p> + +<p>With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.</p> + + Complete Monitors for Gradual Types + + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + 2019-10-31T21:58:26Z + 2019-10-31T21:58:26Z + + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + Four Kinds of Scoping in R + + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-four-kinds-of-scoping-in-r + 2019-09-10T11:00:00Z + 2019-09-10T11:00:00Z + + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">first</a> and <a href="/blog/2019/09/10/scoping-in-r/">second</a> parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.</p> + +<p>In this third and final part of my blog series, I&rsquo;d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.</p> + +<p>This is a &ldquo;bonus&rdquo; blog post, because I&rsquo;m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.</p> +<!-- more--> + +<h2 id="lexical-scope-and-statistical-computation">Lexical scope and statistical computation</h2> + +<p>In <em>Lexical Scope and Statistical Computation</em>,<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">1</a></sup> Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.</p> + +<p>For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:</p> + +<ul> + <li><em>trivial</em>: no free variables allowed</li> + <li><em>static</em>: a free variable takes its value from a set of global variables</li> + <li><em>lexical</em>: a free variable takes the value of the binding that was in effect when the function was defined</li> + <li><em>dynamic</em>: a free variable takes the value of the most recent assignment to that variable</li></ul> + +<p>Note that under this set of definitions, <em>static scoping</em> is a separate scoping rule and not another name for <em>lexical scoping</em>.</p> + +<p>It is possible to simulate each of strategies in R. For fun, we can even construct &ldquo;factories&rdquo; that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)</p> + +<h3 id="template">Template</h3> + +<p>Our examples will follow the template given below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">factory</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="o">&lt;???&gt;</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">factory</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># error, 0, 1, or 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We want to define a <code>factory</code> that takes a function literal and returns a closure that implements the desired scoping rule.</p> + +<p>Our example consists of three definitions of <code>x</code>. On line 5, we assign <code>0</code> to <code>x</code> at the top level. On line 7, we assign <code>1</code> to <code>x</code> inside function <code>h</code>, where we also create the closure. On line 12, we assign <code>2</code> to <code>x</code> inside the function <code>f</code> and right before we call <code>g</code>, which is the closure.</p> + +<p>Finally, we call <code>f</code> and observe the result:</p> + +<ul> + <li>Under trivial scoping, no free variables are allowed, so <code>f()</code> should result in an error.</li> + <li>Under static scoping, free variables may only refer to global variables, so <code>f()</code> should return <code>0</code>.</li> + <li>Under lexical scoping, free variables refer to the variables in scope when the function was defined, so <code>f()</code> should return <code>1</code>.</li> + <li>Under dynamic scoping, free variables take the value from the most recent assignment, so <code>f()</code> should return <code>2</code>.</li></ul> + +<p>We will implement the body of <code>factory</code> in only 3&ndash;5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to <code>factory</code> on line 10.</p> + +<h3 id="trivial-scoping">Trivial scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeTrivial</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">baseenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeTrivial</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># Error in f(0) : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>substitute</code> returns the unevaluated parse tree for <code>fun</code>. In other words, it obtains the literal argument that was passed for <code>fun</code>. This works because of call-by-need semantics in R: function arguments are packaged up into <em>promises</em>. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">2</a></sup> discusses laziness in R in more detail.</p> + +<p>In this example, on line 8, we call <code>factory</code> with <code>function(a) x+a</code> as the argument for the formal parameter <code>fun</code>. Then, we evaluate that parse tree with <code>eval</code>.</p> + +<p>At this point, <code>res</code> is the closure with expression <code>function(a) x+a</code> and a reference to the environment of <code>makeTrivial</code>. On line 3, we change that reference to <code>baseenv()</code>, which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.</p> + +<p>Therefore, variable lookup in the function literal will only search the base environment, so <code>f()</code> results in an error.</p> + +<h3 id="static-scoping">Static scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeStatic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">globalenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeStatic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, on line 3, we update the environment of <code>res</code> to refer to <code>globalenv()</code>, which is the top-level environment where globals are defined.</p> + +<p>Therefore, variable lookup searches the top-level environment, so <code>f()</code> returns <code>0</code>.</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeLexical</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeLexical</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of <code>res</code> to <code>parent.frame()</code>, which is the environment of the function (<code>h</code>) that called the current function (<code>makeLexical</code>). This allows us to simulate lexical scoping, as if the function literal was evaluated inside <code>h</code>, rather than <code>makeLexical</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>h</code>, so <code>f()</code> returns <code>1</code>.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeDynamic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">function</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="nf">res</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> + <span class="p">}</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeDynamic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, we need another level of indirection. <code>makeDynamic</code> returns an anonymous function literal. The anonymous function takes <code>...</code>, which represents an arbitrary list of arguments, and then on line 5 we call <code>res</code> with those exact arguments. Note that we set the environment of <code>res</code> to be the environment of the <em>caller</em> of the anonymous function. Because of the multiple levels of indirection, the caller is <code>f</code>, on line 17.</p> + +<p>On line 12, <code>makeDynamic</code> returns a closure for the anonymous function. <code>h</code> returns that closure when it is called, and assigns it to <code>g</code>. When <code>g</code> is called on line 17, the function literal <code>function(a) x+a</code> is finally evaluated, and its environment is set to the environment of <code>f</code>, the caller of <code>g</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>f</code>, so <code>f()</code> returns <code>2</code>.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the <a href="/blog/2019/09/10/scoping-in-r/">previous post</a>, it&rsquo;s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.</p> + +<p>And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.</p> + +<p><em>I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>A. Goel and J. Vitek. &ldquo;On the Design, Implementation and Use of Laziness in R,&rdquo; in <em>Proceedings of the ACM in Programming Languages (PACMPL)</em>, vol. 3, no. OOPSLA, 2019. To appear. [<a href="http://janvitek.org/pubs/oopsla19a.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">↩</a></p></li></ol></div> + + Scoping in R + + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-scoping-in-r + 2019-09-10T10:00:00Z + 2019-09-10T10:00:00Z + + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">previous post</a> of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: <em>is R lexically or dynamically scoped?</em></p> +<!-- more--> + +<p>Recall the example program from before:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s examine what happens when we run this example. First, we create a mapping for <code>x</code> in the top-level environment. On line 2, we define a function <code>f</code>, which returns the value of some <code>x</code>. On line 3, we define a function <code>g</code>, which creates a new mapping for <code>x</code>, and then calls <code>f</code>. Note that the assignment on line 4 does <em>not</em> update the definition on line 1.</p> + +<p>When <code>f</code> is called, it needs to look up the value of <code>x</code>. In other words, does the reference of <code>x</code> on line 2 refer to the assignment on line 1 or the assignment on line 4? If <code>f</code> returns <code>1</code>, then the behaviour matches lexical scoping. If it returns <code>2</code>, then the behaviour matches dynamic scoping.</p> + +<p>When we run this example, the result is <code>1</code>. This implies that R is lexically scoped.</p> + +<p>But there&rsquo;s more to this story. In the rest of this blog post, I&rsquo;ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.</p> + +<p>The <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">next and final part</a> of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.</p> + +<h2 id="r-is-lexically-scoped-but">R is lexically scoped, but&hellip;</h2> + +<p>In <em>Evaluating the Design of the R Language</em>,<sup><a href="#2019-09-10-scoping-in-r-footnote-1-definition" name="2019-09-10-scoping-in-r-footnote-1-return">1</a></sup> Morandat, Hill, Osvald, and Vitek write:</p> + +<blockquote> + <p>As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.</p></blockquote> + +<p>In other words, R provides many different &ldquo;escape hatches&rdquo;&mdash;ways to bypass lexical scoping. Additionally, even without escape hatches, some of R&rsquo;s functionality can be surprising.</p> + +<h3 id="functions-environments-and-variables-in-r">Functions, environments, and variables in R</h3> + +<p>Before we look at some examples, I think it&rsquo;s useful to briefly discuss some of the core concepts in R that relate to scoping.</p> + +<ul> + <li> + <p><strong>Functions.</strong> R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.</p></li> + <li> + <p><strong>Environments.</strong> An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the &ldquo;enclosing&rdquo; environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.</p></li> + <li> + <p><strong>Variable lookup.</strong> When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.</p></li> + <li> + <p><strong>Variable assignment.</strong> <code>&lt;-</code> is the variable assignment operator in R. The expression <code>x &lt;- 1</code> assigns the value <code>1</code> to the variable <code>x</code> in the current environment. If a mapping for <code>x</code> already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.</p></li></ul> + +<p>From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.<sup><a href="#2019-09-10-scoping-in-r-footnote-2-definition" name="2019-09-10-scoping-in-r-footnote-2-return">2</a></sup></p> + +<p>On the other hand, variable lookup depends on the run-time state of the program&mdash;names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.</p> + +<p>The following examples will make this clear.</p> + +<h3 id="examples">Examples</h3> + +<h4 id="adding-variable-mappings-at-run-time">Adding variable mappings at run time</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it creates a function <code>g</code> that returns <code>x</code>, assigns <code>2</code> to <code>x</code>, and then calls <code>g</code>. When <code>g</code> is called, it looks up <code>x</code>. Since no mapping is found in <code>g</code>&rsquo;s environment, it searches in the enclosing environment (<code>f</code>&rsquo;s), and finds that <code>x</code> has value <code>2</code>. Therefore, <code>g</code> returns <code>2</code>.</p> + +<p>Note that the <code>x</code> on line 3 is resolved only when function <code>g</code> is called, not when it is defined. However, when <code>g</code> is defined, its environment has a reference to <code>f</code>&rsquo;s environment. Therefore, as long as <code>x</code> is defined <em>before</em> <code>g</code> is called, the lookup will always succeed.</p> + +<p>Here&rsquo;s a second example:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>f</code> is a function that branches on its argument, <code>b</code>. If <code>b</code> evaluates to true, then the expression <code>x &lt;- 2</code> is evaluated, and a mapping for <code>x</code> is created in <code>f</code>&rsquo;s environment. Otherwise, no mapping is created.</p> + +<p>When we look up the value of <code>x</code> on line 5, R will first search the function&rsquo;s environment. If <code>b</code> evaluated to true, then R will find a value for <code>x</code>, which is <code>2</code>. Otherwise, R will search in the enclosing environment of <code>f</code>, and find that <code>x</code> is <code>1</code>.</p> + +<p>Both of these examples vaguely resemble dynamic scoping, in that <code>x</code> takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.</p> + +<h4 id="function-lookup">Function lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> <span class="c1"># not an error</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">(</span><span class="m">42</span><span class="p">)</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.</p> + +<p>In this example, we call <code>g</code> with the argument <code>42</code>, which is not a function. Then, in the body of <code>g</code>, we call <code>f(0)</code> on line 3, which requires looking up <code>f</code>. Although there is an <code>f</code> in the environment of <code>g</code>, its value is <code>42</code>, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so <code>f(0)</code> returns <code>0</code>.</p> + +<p>This behaviour exists because <code>c</code> is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.</p> + +<h4 id="super-assignment">Super assignment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="n">x</span> <span class="o">&lt;&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +<span class="n">x</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>&lt;&lt;-</code> is the &ldquo;super assignment&rdquo; operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.</p> + +<p>In the above program, we define <code>x</code> to be <code>0</code> at the top level, and then define the function <code>f</code>. When we call <code>f</code> on line 7, it assigns <code>1</code> to <code>x</code> on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, <code>f</code> returns <code>x</code>, which is looked up from the local environment and has value <code>1</code>. Finally, line 8 looks up <code>x</code> from the top level environment, which has value <code>2</code>.</p> + +<h4 id="evaluating-arbitrary-code">Evaluating arbitrary code</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">eval</span><span class="p">(</span><span class="nf">parse</span><span class="p">(</span><span class="n">text</span> <span class="o">=</span> <span class="n">t</span><span class="p">))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="s">"x &lt;- 0"</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument <code>t</code>, which happens to be the string <code>"x &lt;- 0"</code>. Then, when line 4 executes, the lookup of <code>x</code> returns <code>0</code>.</p> + +<h4 id="simulating-dynamic-scope">Simulating dynamic scope</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.frame</span><span class="p">())</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 3, we perform an explicit variable lookup for <code>x</code>, but we do so in the environment <code>parent.frame()</code>, which refers to the calling function&rsquo;s environment, in this case, <code>g</code>&rsquo;s environment.. Therefore, the lookup returns <code>2</code>.</p> + +<p>Note that R has a similarly named function, <code>parent.env(e)</code> which returns the <em>enclosing environment</em> of the given environment <code>e</code>.</p> + +<h4 id="constructing-an-arbitrary-environment">Constructing an arbitrary environment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">e</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> + <span class="n">e</span><span class="o">$</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">3</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="n">e</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 3</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it constructs a new environment, <code>e</code>, which is initially empty. (By default, its enclosing environment is the current environment, which is <code>f</code>&rsquo;s.) Next, on line 4, it directly adds a mapping to that environment, assigning <code>3</code> to <code>x</code>. Then, on line 5, the lookup is explicitly done in environment <code>e</code>, so <code>f</code> returns <code>3</code>.</p> + +<h4 id="deleting-mappings">Deleting mappings</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">rm</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.env</span><span class="p">(</span><span class="nf">environment</span><span class="p">()))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># Error in f() : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Not only is it possible to dynamically add and modify mappings in R, but it is also possible to <em>delete</em> mappings. This is what line 3 does: it explicitly removes the mapping for <code>x</code> from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when <code>f</code> is called, the lookup of <code>x</code> fails and an error is raised.</p> + +<h4 id="infinite-loop-during-variable-lookup">Infinite loop during variable lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">enva</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="n">envb</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">enva</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">envb</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">envb</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="nf">environment</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="nf">f</span><span class="p">()</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.</p> + +<p>On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: <code>enva</code>&rsquo;s enclosing environment is <code>envb</code>, and <code>envb</code>&rsquo;s enclosing environment is <code>enva</code>.</p> + +<p>On line 5, we define a function with a free variable, <code>x</code>, but on line 6, we set <code>f</code>&rsquo;s environment to be <code>enva</code>. Finally, we call <code>f</code>.</p> + +<p>When the body of <code>f</code> is evaluated, it needs to look up <code>x</code>. Lookup starts in <code>f</code>&rsquo;s environment, which we set to be <code>enva</code>. Since no mapping for <code>x</code> is found, lookup continues in <code>enva</code>&rsquo;s enclosing environment, which is <code>envb</code>. However, <code>envb</code> is also empty, so lookup continues in its enclosing environment, which is <code>enva</code>, and now lookup results in an infinite loop.</p> + +<h3 id="an-intuition-for-scoping-in-r">An intuition for scoping in R</h3> + +<p>Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1"># example 1</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> + +<span class="c1"># example 2</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>It seems that <code>x</code> takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What&rsquo;s missing from our intuition?</p> + +<p>The key insight is that R is <em>function scoped</em>. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <code>&lt;-</code> <em>does not create a new scope</em>, and it is more useful to think of it as a mutation <em>on the current environment</em>. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)</p> + +<p>In a sense, it might be more accurate to say that R <em>environments</em> are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.</p> + +<h2 id="discussion">Discussion</h2> + +<p>All of this might make you a little uncomfortable, and uncertain about R&rsquo;s scoping rules.</p> + +<p>On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply &ldquo;lexical-like&rdquo; behaviour, and the creators have confirmed that lexical scoping was the intent.</p> + +<p>On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment&mdash;but this is not consistent with R&rsquo;s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>This ambiguity shows up in a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-3-definition" name="2019-09-10-scoping-in-r-footnote-3-return">3</a></sup> where the authors write:</p> + +<blockquote> + <p>Furthermore, because variable scoping in R is dynamic and can be modified at the language level [&hellip;] it cannot be trivially guaranteed that <code>x</code> is going to point to the same data structure throughout the entire execution of the loop.</p></blockquote> + +<p>It is true that a variable <code>x</code> may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.</p> + +<p>It is true that variable <em>lookup</em> is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of <em>dynamic scope</em>, then it would be fair to say that R is dynamically scoped.</p> + +<p>But if your definition of <em>dynamic scope</em> is &ldquo;a variable is bound to the most recent assignment during the program&rsquo;s execution,&rdquo; then it is not correct to say R is dynamically scoped.</p> + +<p>I think we have this ambiguity because <em>scope</em> (the places in a program where a variable can be referenced) and <em>variable lookup</em> or <em>name resolution</em> (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.</p> + +<p>Personally, I prefer the definition of <em>scope</em> that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is <em>both</em> lexically and dynamically scoped, or that R is <em>neither</em> lexically and dynamically scoped.</p> + +<p>I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language&mdash;when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.</p> + +<p>Regardless, it is very easy to miscommunicate, so I think it&rsquo;s important to be very clear and make sure you and your audience know what definitions of scoping you&rsquo;re using!</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>This entire adventure started when we were working on a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-4-definition" name="2019-09-10-scoping-in-r-footnote-4-return">4</a></sup> and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!</p> + +<p>This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.</p> + +<p>To summarize, I would say that <em>scope</em> refers to the places in a program where a variable is visible and can be referenced. Under <em>lexical scoping</em>, the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program. Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment during the program&rsquo;s execution.</p> + +<p>I would say that R <em>aims</em> to be lexically scoped&mdash;it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!</p> + +<p>If you want more content about R and scoping, the <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">third and final part</a> of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.</p> + +<p><strong>Edited 2020/02/21:</strong> For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), <a href="http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/">this blog post</a> has some nice examples and diagrams.</p> + +<p><em>I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.</em></p> + +<p><em>If you liked this post, you may also be interested in the following Twitter threads about R: <a href="https://twitter.com/mhyee/status/1063983175163158531">one</a>, <a href="https://twitter.com/mhyee/status/1067818720532316166">two</a> and <a href="https://twitter.com/mhyee/status/1074744049951739905">three</a>.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>F. Morandat, B. Hill, L. Osvald, J. Vitek. &ldquo;Evaluating the Design of the R Language,&rdquo; in <em>Proceedings of the European Conference on Object-Oriented Programming (ECOOP)</em>, 2012. [<a href="https://doi.org/10.1007/978-3-642-31057-7_6">DOI</a>][<a href="http://janvitek.org/pubs/ecoop12.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. &ldquo;Lexical Scope and Statistical Computing&rdquo;, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-2-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-3-definition" class="footnote-definition"> + <p>L. Stadler, A. Welc, C. Humer, and M. Jordan. &ldquo;Optimizing R Language Execution via Aggressive Speculation,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2016. [<a href="https://doi.org/10.1145/2989225.2989236">DOI</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-3-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-4-definition" class="footnote-definition"> + <p>O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. &ldquo;R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2019. To appear. [<a href="http://janvitek.org/pubs/dls19.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-4-return">↩</a></p></li></ol></div> + + Lexical and Dynamic Scope + + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + 2019-09-05T10:00:00Z + 2019-09-05T10:00:00Z + + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> + + [Conversational Concurrency (cross-post)](https://eighty-twenty.org/2018/01/24/conversational-concurrency) + + urn:http-prl-ccs-neu-edu:-blog-2019-05-11-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency + 2019-05-11T00:03:16Z + 2019-05-11T00:03:16Z + + Tony Garnock-Jones + + + Forgetful and Heedful contracts + + urn:http-prl-ccs-neu-edu:-blog-2019-04-07-forgetful-and-heedful-contracts + 2019-04-07T23:15:11Z + 2019-04-07T23:15:11Z + + Ben Greenman + +<p><em>Forgetful</em> and <em>heedful</em> are two methods for space-efficient contracts developed by <a href="http://www.cs.pomona.edu/~michael/">Michael Greenberg</a> in <a href="https://arxiv.org/abs/1410.2813">2014</a>. These methods were born in the shadow of a third method, <em>eidetic</em>, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.</p> +<!-- more--> + +<hr /> + +<p>Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of <a href="https://www2.ccs.neu.edu/racket/pubs/popl11-dfff.pdf">three parties</a>: the API code, the client code, and the contract between them.</p> + +<p>For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:</p> + +<pre><code> #lang racket/base + + (module sort racket + (provide + (contract-out + [quicksort + (-&gt; (vectorof point/c) void?)])) + + (define point/c (vectorof integer?)) + + (define (quicksort points) + ....)) + + (module client racket + (require (submod ".." sort)) + (quicksort '())) + + (require 'client)</code></pre> + +<pre><code>quicksort: contract violation; + expected a vector + given: '() + in: the 1st argument of + (-&gt; (vectorof (vectorof integer?)) void?) + contract from: + (file.rkt sort) + blaming: (file.rkt client) + (assuming the contract is correct)</code></pre> + +<p>That covers the basics. For an extended introduction to contracts, visit <a href="https://docs.racket-lang.org/guide/contracts.html">The Racket Guide</a>.</p> + +<p>The quicksort example and the related figures are from the paper <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts: Fixing a Pathology of Gradual Typing</em></a></p> + +<h3 id="classic-contracts-and-space-efficiency">Classic contracts and &ldquo;Space Efficiency&rdquo;</h3> + +<p>The <code>(vectorof point/c)</code> contract used above describes a possibly-mutable array whose elements match the <code>point/c</code> contract. Since the array can be mutated, this contract has implications for two parties:</p> + +<ol> + <li>the client module must supply a good array, and</li> + <li>the sorting module must not insert a bad element.</li></ol> + +<p>To enforce the second condition, the <code>vectorof</code> contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:</p> + +<pre><code>(quicksort (vector (vector 4 4) + (vector 2 2) + (vector 1 1) + (vector 3 3)))</code></pre> + +<p>After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a <strong>solid</strong> blue bar for the <strong>write</strong> checks against the sort module and a <em>striped</em> blue bar for the <em>read</em> checks against the client.</p> + +<p><img src="/img/vector-chaperone-0.png" alt="A wrapped vector" /></p> + +<p>In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted &mdash; because every <strong>write</strong> and <em>read</em> applies a contract to the element.</p> + +<p><img src="/img/vector-chaperone-1.png" alt="Layers of element wrappers" /></p> + +<p>On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.</p> + +<p>Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and &ldquo;indirecting&rdquo; future writes/reads through wrappers. These space and time costs can add up.</p> + +<blockquote> + <p>&ldquo;on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times&rdquo; &mdash; <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts</em></a></p></blockquote> + +<p>To fix the problem, researchers have been exploring <em>space-efficient</em> implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and <em>eidetic</em>, <em>forgetful</em>, and <em>heedful</em> are his names for three implementations.</p> + +<p>(Although the goal of this post is to promote <em>forgetful</em> and <em>heedful</em>, we will review all three.)</p> + +<h3 id="eidetic-space-efficiency">Eidetic space-efficiency</h3> + +<p>The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a <em>merge</em> operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal &ldquo;wrapping&rdquo; contracts and their size is bounded by the number (and height) of source-code contracts in the program.</p> + +<p>An eidetic contract is an <code>N</code>-ary tree (for <code>N &gt; 0</code>):</p> + +<ul> + <li>each node represents a higher-order contract combinator, such as <code>vectorof</code></li> + <li>the <code>N</code> children of a node represent the different interactions that the value supports</li> + <li>each leaf is a list of non-higher-order, or <em>flat</em>, contracts</li></ul> + +<p>For example, the <code>(vectorof point/c)</code> source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf">Collapsible Contracts</a> paper has an illustration. Each tree node represents a <code>vectorof</code> contract; these nodes have <code>N=2</code> children because vectors support reads and writes.</p> + +<p>A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists &mdash; if an implementation comes with a <code>contract-stronger?</code> predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.</p> + +<p>Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should <code>contract-stronger?</code> try to solve problems that lack polynomial-time solutions?</p> + +<p>Thankfully, there are at least two &ldquo;compromise&rdquo; alternatives.</p> + +<h3 id="forgetful-space-efficiency">Forgetful space-efficiency</h3> +<!-- "no operation relies on e being a T2, skipping the check doesn't risk soundness" p.12--> +<!-- "In forgetful \lambda_H, we offer a simple solution to space inefficient casts: just forget about them" p.11--> +<!-- "Just the same, when accumulating casts on the stack, we throw away all but the last cast" p.11--> +<!-- "forgetful ... skip[s] checks and change[s] blame labels" p.3--> + +<blockquote> + <p>&ldquo;Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 10, bottom of page 23--></blockquote> + +<p>The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:</p> + +<ol> + <li>the client obligations from the old contract, and</li> + <li>the server obligations from the new contract</li></ol> + +<p>If not, proceed as usual &mdash; by wrapping (an unwrapped value) or raising an error. Every value receives at most <strong>one</strong> wrapper; this wrapper changes as the value flows to different clients.</p> + +<p>Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module <code>A</code> exports a function <code>f</code> with contract <code>(-&gt; T1 T2)</code> to module <code>B</code>, and suppose module <code>B</code> shares this function with a few other client modules using different contracts. As <code>f</code> flows to a new client, it keeps the <code>T1</code> domain check and gets a replacement for the <code>T2</code> codomain check.</p> + +<ul> + <li>Keeping <code>T1</code> ensures that the code inside the function (defined by module <code>A</code>) receives input that matches its expectation.</li> + <li>Replacing <code>T2</code> ensures that each new client receives output that it expects.</li></ul> + +<p>Unfortunately, replacing <code>T2</code> also means that clients of module <code>B</code> cannot trust the <code>T2</code> contract. This contract is not checked, and so forgetful contracts <strong>miss</strong> some errors that would be caught by standard contracts. For the same reason, a bug in module <code>B</code> may go undetected by its clients &mdash; even if a later contract reports an issue, the contract system has no memory that <code>B</code> was partly-responsible.</p> + +<p>Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.</p> + +<h3 id="heedful-space-efficiency">Heedful space-efficiency</h3> + +<p>A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set &mdash; or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.</p> + +<p>To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.</p> + +<p>The heedful method is a compromise between forgetful and eidetic.</p> + +<ul> + <li> + <p>Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of <code>contract-stronger?</code> predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.</p></li> + <li> + <p>Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.</p></li></ul> + +<p>For details, see <a href="https://arxiv.org/abs/1410.2813">the extended version</a> of Michael&rsquo;s POPL 2015 paper. Don&rsquo;t bother searching <a href="http://www.cs.pomona.edu/~michael/papers/popl2015_space.pdf">the conference version</a> &mdash; aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.</p> + +<p><code>*</code> If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to <a href="http://mballantyne.net/">Michael Ballantyne</a> for explaining this to me.</p> + +<h3 id="priorities-and-appearances">Priorities and Appearances</h3> + +<p>The extended version of <em>Space-Efficient Manifest Contracts</em> introduces the forgetful and heedful methods with extreme modesty. It&rsquo;s tempting to skip past them and focus on the eidetic method.</p> + +<blockquote> + <p>&ldquo;Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [<a href="https://dl.acm.org/citation.cfm?doid=1706299.1706342">Siek and Wadler 2010</a>]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen&mdash;warm up exercises for eidetic.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 1, bottom of page 2--></blockquote> + +<p>And yet, at least two other research papers rely on these &ldquo;strawmen&rdquo; &mdash; or rather, the ideas behind the names.</p> + +<p><a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a>, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.</p> + +<blockquote> + <p>&ldquo;if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- page 21--></blockquote> + +<p>This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.</p> + +<blockquote> + <p>&ldquo;removing these casts preserves the soundness of the evaluation while reducing the number of them&rdquo;</p> + <p>&ldquo;while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- p.27--><!-- page 21--></blockquote> +<!-- The followup at POPL 2019 is not forgetful.--> +<!-- It's similar to eager coercions ... keep all types around and error--> +<!-- if there's a new type that doesn't match the old ones.--> +<!-- Also, that paper chooses not to let functions have intersection types,--> +<!-- which kind-of-avoids the questions ... but really the eagerness is key.--> + +<p><a href="https://dl.acm.org/citation.cfm?id=3009849"><em>Big Types in Little Runtime</em></a>, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their <em>transient</em> semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.</p> + +<p>That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value&rsquo;s set. When a mismatch occurs, the semantics <a href="https://www.ccs.neu.edu/home/types/resources/notes/transient-undefined-blame-extract.pdf">tries to report</a> every type in the set that relates to the mismatch.</p> + +<p>And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">ICFP 2018</a>. Where will we see them next?</p> + + PLISS: Learn About PL Implementation in a Castle + + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + 2019-03-09T14:40:16Z + 2019-03-09T14:40:16Z + + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> + + Writing a paper with Scribble + + urn:http-prl-ccs-neu-edu:-blog-2019-02-17-writing-a-paper-with-scribble + 2019-02-17T16:20:50Z + 2019-02-17T16:20:50Z + + Ben Greenman + +<p>This post explains how to get started using Scribble to write a research paper.</p> +<!-- more--> + +<hr /> + +<blockquote> + <p>This post was written using <a href="http://download.racket-lang.org/all-versions.html">Racket 7.1</a> and <a href="https://github.com/racket/scribble/releases/tag/v7.1">Scribble 1.29</a></p></blockquote> + +<p>Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:</p> + +<ul> + <li>the paper can import definitions from the research, keeping a single point of control;</li> + <li>the language&rsquo;s functional abstractions can help manage the writing;</li> + <li>the language&rsquo;s drawing and/or plotting libraries can replace <a href="https://ctan.org/pkg/pgf?lang=en">TikZ</a>;</li> + <li>and you can write unit tests to validate the claims made in the paper.</li></ul> + +<p>Scribble, <a href="http://docs.racket-lang.org/scribble/index.html">the Racket documentation tool</a>, comes with a to-LaTeX compiler and a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> library tailored to the new <a href="https://ctan.org/pkg/acmart?lang=en">ACM paper format</a>. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short &ldquo;getting started&rdquo; guide and links to some larger examples.</p> + +<blockquote> + <p>For a Scribble tutorial, see the links in: <a href="/blog/2017/05/23/building-a-website-with-scribble/index.html">Building a Website with Scribble</a></p></blockquote> + +<h2 id="getting-started-with-">Getting started with <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a></h2> + +<p>The first line of a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> document sets the formatting options (similar to a LaTeX file using <code>acmart.cls</code>). For example, the <a href="https://conf.researchr.org/track/gpce-2018/gpce-2018#Call-for-Papers">GPCE 2018 call for papers</a> asks for anonymized <code>sigplan</code>-format submissions with line numbers and 10 point font. The proper Scribble incantation is:</p> + +<pre><code>#lang scribble/acmart @sigplan @anonymous @review @10pt</code></pre> + +<p>Next, you may want to import some definitions. If we have a file <code>references.rkt</code> (see below for a definition), we can import it as follows:</p> + +<pre><code>@require{references.rkt}</code></pre> + +<p>The third main ingredient is the title and author information:</p> + +<pre><code>@(define neu (affiliation #:institution "Northeastern University")) +@(define anon (email "anon@anon.net")) + +@title{Writing a paper with Scribble} +@author[#:affiliation neu #:email anon]{Ben Greenman} + +@; optional: set the author names in the page headers +@elem[#:style "Sshortauthors"]{B. Greenman}</code></pre> + +<p>The paper is now ready to be written. You can forge ahead with a new <a href="http://docs.racket-lang.org/scribble/base.html#%28def._%28%28lib._scribble%2Fbase..rkt%29._section%29%29">section</a> and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and <a href="http://docs.racket-lang.org/scribble/base.html#%28form._%28%28lib._scribble%2Fbase..rkt%29._include-section%29%29">import</a> content from other modules:</p> + +<pre><code>@include-abstract{abstract.scrbl} +@include-section{introduction.scrbl}</code></pre> + +<p>Finally, the main page is a good place to <a href="https://docs.racket-lang.org/scriblib/autobib.html">generate the bibliography</a>. Assuming this document imports a file like the <code>references.rkt</code> below, this expression inserts a bibliography titled &ldquo;References&rdquo;:</p> + +<pre><code>@generate-bibliography[#:sec-title "References"]</code></pre> + +<p>To build the document, invoke <code>scribble</code> on the command-line with the <code>--pdf</code> or <code>--latex</code> options:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl</code></pre> + +<p>If all goes well, this command generates a <code>FILE.pdf</code> with properly-linked cross references.</p> + +<h3 id="auxiliary-files">Auxiliary Files</h3> + +<p>If you save the code above to a file <code>example.scrbl</code> and save the files below in the same directory, then you should be able to build an <code>example.pdf</code>.</p> + +<p>These files are available in a slightly different format at this link:</p> + +<ul> + <li><a href="https://gitlab.com/bengreenman/scribble-acmart-example">https://gitlab.com/bengreenman/scribble-acmart-example</a></li></ul> + +<h4 id="referencesrkt"><code>references.rkt</code></h4> + +<pre><code>#lang racket/base + +(provide + ~cite citet generate-bibliography + fbf-icfp-2009) + +(require + scriblib/autobib) + +(define-cite ~cite citet generate-bibliography + #:style author+date-square-bracket-style) + +(define icfp "ICFP") + +(define fbf-icfp-2009 + (make-bib + #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools" + #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler") + #:location (proceedings-location icfp #:pages '(109 120)) + #:date 2017))</code></pre> + +<h4 id="abstractscrbl"><code>abstract.scrbl</code></h4> + +<pre><code>#lang scribble/acmart + +A simple Scribble document.</code></pre> + +<h4 id="introductionscrbl"><code>introduction.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require{references.rkt} + +@; start with `title` instead of `section`, because importing via +@; `include-section` shifts all title/section/subsections down one level +@title{Introduction} + +Scribble creates a connection between a stand-alone document and the artifact +it describes@~cite[fbf-icfp-2009].</code></pre> + +<h3 id="q-how-to-debug-scribble-error-messages">Q. How to debug Scribble error messages?</h3> + +<p>If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.</p> + +<p>As a compile-time example, adding <code>@ foo</code> to a document produces the message <code>unexpected whitespace after @</code> and you can either delete the whitespace or change the <code>@</code> to <code>@"@"</code> for a literal <code>@</code>-sign.</p> + +<p>As a run-time example, adding <code>@(+ 2 2)</code> produces this message:</p> + +<pre><code>not valid in document body (need a pre-part for decode) in: 4</code></pre> + +<p>One fix is to convert <code>4</code> to a string, as in <code>@~a[(+ 2 2)]</code>.</p> + +<p>But if something goes wrong when Scribble renders a generated document to PDF, the default error output is <strong>not</strong> likely to help. For example, adding <code>@elem[#:style "oops"]</code> to a document produces a giant message:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl +[[ ... 84K of output ... ]] +Output written on example.pdf (1 page, 277876 bytes). +PDF statistics: + 53 PDF objects out of 1000 (max. 8388607) + 37 compressed objects within 1 object stream + 7 named destinations out of 1000 (max. 500000) + 36877 words of extra memory for PDF output out of 42996 (max. 10000000) + +run-pdflatex: got error exit code + context...: + [[ ... 17 more lines ... ]]</code></pre> + +<p>The best way to debug these messages is to <strong>ignore them</strong> and use a LaTeX compiler directly. For the &ldquo;oops&rdquo; mistake, LaTeX stops at the undefined control sequence &mdash; giving a hint about how to find the problem:</p> + +<pre><code>$ raco scribble --latex FILE.scrbl +$ pdflatex FILE.tex +[[ ... 12KB of output ... ]] +! Undefined control sequence. +l.549 \oops + {} +? </code></pre> + +<h3 id="q-how-to-add-a-latex-style-file">Q. How to add a LaTeX style file?</h3> + +<p>To add extra LaTeX code to the final document, create a new file and include it with the <code>++style</code> command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).</p> + +<pre><code>$ raco scribble ++style style.tex --pdf FILE.scrbl</code></pre> + +<p>Here is an example style file.</p> + +<h4 id="styletex"><code>style.tex</code></h4> + +<pre><code>\settopmatter{printfolios=true,printccs=true,printacmref=true} +% add page numbers etc. + +\overfullrule=1mm +% draw a black rectangle near lines that overflow the margin</code></pre> + +<p>Another way to add extra LaTeX code is to add a <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Flatex-properties..rkt%29._tex-addition%29%29"><code>tex-addition</code></a> style property to the main title. This second approach makes it easy to include more than one file:</p> + +<pre><code>#lang scribble/acmart + +@require[ + (only-in scribble/core make-style) + (only-in scribble/latex-properties make-tex-addition)] + +@(define extra-style-files + (list (make-tex-addition "style.tex"))) + +@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble} + +@; ....</code></pre> + +<h3 id="q-how-to-make-a-figure">Q. How to make a figure?</h3> + +<p>Use the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library to add figures to a document.</p> + +<pre><code>@require[pict scriblib/figure] +@figure[ + "fig:fish" @; figure tag, see `figure-ref` + @elem{A Standard Fish} @; figure caption, appears below the content + @elem{fish = @(standard-fish 90 40)}] @; content</code></pre> + +<p>The content of a figure can be almost anything that would work in the toplevel of the document.</p> + +<h3 id="q-how-to-include-extra-files-pictures-latex">Q. How to include extra files (pictures, LaTeX)?</h3> + +<p>The <code>++extra</code> command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.</p> + +<p>For example, if a document includes the content of an external LaTeX file:</p> + +<pre><code>@elem[#:style "input"]{inline-this.tex}</code></pre> + +<p>then make sure to build the document with a command like this:</p> + +<pre><code>$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl</code></pre> + +<h4 id="inline-thistex"><code>inline-this.tex</code></h4> + +<pre><code>% Raw LaTeX allowed here +$\lambda x.\, x$</code></pre> + +<h3 id="q-what-about-in-line-latex">Q. What about in-line LaTeX?</h3> + +<p>An <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Fcore..rkt%29._element%29%29">element</a> with the <a href="https://docs.racket-lang.org/scribble/core.html#%28idx._%28gentag._60._%28lib._scribblings%2Fscribble%2Fscribble..scrbl%29%29%29"><code>'exact-chars</code></a> <a href="https://docs.racket-lang.org/scribble/core.html#%28tech._style._property%29">style property</a> renders directly to LaTeX.</p> + +<pre><code>@(define (exact . stuff) + @; the style name "relax" puts a `\relax` no-op in front of the stuff + (make-element (make-style "relax" '(exact-chars)) stuff)) + +@exact|{$\lambda x.\, x$}| +@; ==&gt; \relax{$\lambda x.\, x$} + +@(define ($ . math-stuff) + (apply exact (list "$" math-stuff "$"))) + +@${\lambda x.\, x} +@; ==&gt; \relax{$\lambda x.\, x$}</code></pre> + +<h2 id="creating-a-httpdocsracket-langorgguidemodulesyntaxhtml28parthash-lang29lang-for-a-paper">Creating a <a href="http://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29">#lang</a> for a paper</h2> + +<p>For a Scribble document that is split across multiple files, it can be helpful to make a <code>#lang</code> that <a href="http://blog.racket-lang.org/2017/03/languages-as-dotfiles.html">provides a common environment</a>. Instead of starting each file with a <code>require</code>, e.g.:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require["references.rkt" "helper-functions.rkt" scriblib/figure] + +....</code></pre> + +<p>files can start with a name that describes their common purpose:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang conference-2018-submission + +....</code></pre> + +<p>As a bonus, if the language is defined as a package then the Scribble document can use Racket&rsquo;s dependency management tools:</p> + +<pre><code># to install the paper and interactively install dependencies: +$ cd conference-2018-submission; +$ raco pkg install + +# To check that the paper builds with no dependency issues: +$ raco setup --check-pkg-deps conference-2018-submission + +# To run all unit tests +$ raco test -c conference-2018-submission</code></pre> + +<p>To create a package and language:</p> + +<ol> + <li>Move the Scribble document to a directory with the language name, i.e., <code>conference-2018-submission/</code></li> + <li>Write a simple <code>info.rkt</code> to configure the package</li> + <li>Create a normal Racket module that exports the common environment</li> + <li>Create a <code>conference-2018-submission/lang/reader.rkt</code> module</li></ol> + +<p>Details below. For a full example, visit:</p> + +<ul> + <li><a href="https://gitlab.com/bennn/scribble-acmart-example">https://gitlab.com/bennn/scribble-acmart-example</a></li></ul> + +<hr /> + +<h4 id="conference-2018-submissioninforkt"><code>conference-2018-submission/info.rkt</code></h4> + +<p>This file defines the basic metadata for a package. For more about <code>info.rkt</code>, see: <a href="http://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html">Tutorial: Creating a Package</a>.</p> + +<pre><code>#lang info +(define collection "conference-2018-submission") +(define deps '("base" "scribble-lib" "at-exp-lib")) +(define build-deps '("racket-doc" "scribble-doc")) +(define pkg-desc "Paper for Conference 2018") +(define version "0.1")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionmainrkt"><code>conference-2018-submission/main.rkt</code></h4> + +<p>This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> language, the file &ldquo;references.rkt&rdquo;, and the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library.</p> + +<pre><code>#lang racket/base + +(provide + (all-from-out + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")) + +(require + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionlangreaderrkt"><code>conference-2018-submission/lang/reader.rkt</code></h4> + +<p>This file: (1) tells Racket to use the Scribble reader on <code>#lang conference-2018-submission</code> modules, and (2) wraps the result of such modules in a shape that Scribble expects.</p> + +<pre><code>#lang s-exp scribble/base/reader +conference-2018-submission +#:wrapper1 (lambda (t) (cons 'doc (t)))</code></pre> + +<h2 id="links-to-example-documents">Links to Example Documents</h2> + +<p>These documents use the <code>#lang</code> approach to writing a paper with Scribble. Check their <code>main.rkt</code> for example formatting functions and unit tests, and check the <code>.scrbl</code> files to see how the ideas above look in a larger document.</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018">https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018</a></li> + <li><a href="https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018">https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018</a></li></ul> + +<p>Finally, this repository provides a tool to start a new Scribble document:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gtp-paper">https://pkgd.racket-lang.org/pkgn/package/gtp-paper</a></li></ul> + +<h2 id="further-reading">Further Reading</h2> + +<ul> + <li><a href="https://project.inria.fr/coqexchange/checking-machine-checked-proofs/">Checking Machine-Checked Proofs</a></li></ul> + + On-Stack Replacement + + urn:http-prl-ccs-neu-edu:-blog-2019-01-28-on-stack-replacement + 2019-01-28T10:29:57Z + 2019-01-28T10:29:57Z + + Ming-Ho Yee + +<p>Last semester, I took <a href="https://course.ccs.neu.edu/cs7600/">a course</a> where the final project was to write a survey paper on &ldquo;a topic in the intersection between computer systems and your area.&rdquo; So I wrote about on-stack replacement.</p> +<!-- more--> + +<p><strong>Abstract</strong></p> + +<blockquote> + <p>On-stack replacement (OSR) is a programming language implementation technique that allows a running program to switch to a different version of code. For example, a program could start executing optimized code, and then transfer to and start executing unoptimized code. This was the original use case for OSR, to facilitate debugging of optimized code.</p> + <p>After its original use was established, OSR shifted to a different use case: optimizing programs. OSR allows the run-time system to detect if a program is executing an inefficient loop, recompile and optimize the method that contains the loop, and then transfer control to the newly compiled method. Another strategy is to optimize code based on some assumptions, then, if the assumptions are invalidated at run-time, transfer control back to the original, unoptimized code.</p> + <p>In this survey paper, we study how OSR was first introduced as a means for debugging, how it came to be used for program optimizations, its implementation as a reusable library, and other directions of research.</p></blockquote> + +<p>If you&rsquo;re interested, you can find a copy <a href="/img/cs7600-mhyee-survey-paper-osr.pdf">here</a> or on <a href="https://www.overleaf.com/read/smcmsnksxfdk">Overleaf</a>.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in <a href="http://prl.ccs.neu.edu/blog/2017/03/15/tracing-jits-for-dynamic-languages/">tracing JITs for dynamic languages</a>.</em></p> + + The Behavior of Gradual Types: A User Study + + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + 2018-12-11T19:50:33Z + 2018-12-11T19:50:33Z + + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + Java and Migratory Typing + + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + 2018-12-02T14:41:53Z + 2018-12-02T14:41:53Z + + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> + + Turnstile Mailing List + + urn:http-prl-ccs-neu-edu:-blog-2018-11-30-turnstile-mailing-list + 2018-11-30T14:55:30Z + 2018-11-30T14:55:30Z + + Stephen Chang + +<p><a href="https://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> now has a mailing list: <a href="https://groups.google.com/forum/#!forum/turnstile-users">https://groups.google.com/forum/#!forum/turnstile-users</a></p> +<!-- more--> + +<p>There is also a <code>#turnstile</code> channel on <a href="https://racket.slack.com">the Racket Slack</a>.</p> + + Disappearing Code + + urn:http-prl-ccs-neu-edu:-blog-2018-11-24-disappearing-code + 2018-11-24T09:52:58Z + 2018-11-24T09:52:58Z + + Ben Greenman + +<p>Two experiences at <a href="https://2018.splashcon.org/home">SPLASH 2018</a> reminded me that software gets thrown away and replaced.</p> +<!-- more--> + +<h3 id="story-1">Story 1</h3> + +<p>The first reminder came near the end of a <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a> by <a href="https://people.csail.mit.edu/rinard/">Martin Rinard</a>. Once upon a time, Martin was working as a consultant and a firm asked him to review a software package. (The firm wanted a second opinion about how the software computed its results.) The firm sent a zipfile; Martin found six versions of the code inside; the firm said &ldquo;well, please check all six versions&rdquo;; and it turned out:</p> + +<ul> + <li><strong>Version 1</strong> : the source code was written in a domain-specific language (DSL) that generated code for the application</li> + <li><strong>Version 2</strong> : the DSL source was the same as version 1, but the generated code was slightly modified</li> + <li>&hellip;</li> + <li><strong>Version 6</strong> : the generated code was the source code and the DSL was gone</li></ul> + +<p>The moral of Martin&rsquo;s story was: (1) the creators of a software system are often different from the maintainers, and (2) researchers need to build tools to help these maintainers.</p> + +<h3 id="story-2">Story 2</h3> + +<p>The second reminder came from a teaching assistant who said the <a href="https://www.cs.cornell.edu/courses/cs3110/2018fa/">functional programming course</a> at their institution was currently using a Python script to test students&rsquo; code. Once upon a time, I was a teaching assistant for the <a href="https://www.cs.cornell.edu/courses/cs3110/2014sp/">same course</a> at the same institution. We had trouble testing students&rsquo; code via the Python script left by the pre&ndash;2013 course staff, so I wrote a <a href="https://gitlab.com/bengreenman/ocaml_tools/">command-line tool</a> to handle the tests and other compile/run/grade tasks. To keep history from repeating itself, I used the same language the course teaches (OCaml) and wrote some documentation &mdash; but it seems like that was not enough. At any rate, writing the tool was a good exercise.</p> + +<blockquote> + <p><em>In the end, everybody must understand for himself.</em> &mdash; <a href="https://dl.acm.org/citation.cfm?id=3731">Per Martin-Löf</a></p></blockquote> + +<h3 id="reflection">Reflection</h3> + +<p>In each story, the maintainers of a software system threw away some old code to make their job easier in the short term. How can we stop this &ldquo;re-inventing the wheel&rdquo; from happening?</p> + +<p>Martin Rinard&rsquo;s solution is to let maintenance programmers keep their current habits, but provide tools to make the short-term, pragmatic solutions into a more robust systems. Search for "<a href="https://people.csail.mit.edu/rinard/paper/osdi04.pdf">failure-oblivious computing</a>" to learn more (this was the topic of his <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a>).</p> + +<p>In Story 1, the maintainers were able to avoid the DSL by modifying an inherited blob of DSL-generated code. If the DSL did not generate code, history might have taken a different course; it might be best to start with a language that offers tools for linguistic re-use, and to build a DSL from these tools &mdash; so there is no generated code. The Racket programming language is exploring this path. For a recent example, see the <a href="https://www2.ccs.neu.edu/racket/pubs/icfp17-acf.pdf">video-lang paper</a>.</p> + +<p>The Story 2 test harness, however, was not generating code. Its maintainers discarded a &ldquo;big&rdquo; program written in a typed functional language in favor of a script. Perhaps we need a language that allows mixing statically-typed and dynamically-typed code (shouts out to <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">my own research</a>).</p> + +<p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p> + +<hr /> + +<h4 id="addendum-comment-from-mitch-wand">Addendum: comment from Mitch Wand</h4> + +<blockquote> + <p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p></blockquote> + +<p>Ermm, this works better for sourdough bread than for people.</p> + +<p>Even in the not-so-real world of checking student solutions, there&rsquo;s often no way of guaranteeing that one half of a pair will be around for the second round. They may be on co-op. Or the course will not be offered the next semster/year/etc. Or the course will change at the next offering (from OCaml to Python or from Racket to Java) so that large chunks of the infrastructure will have to be discarded or rewritten.</p> + +<p>The &ldquo;real&rdquo; solution is to write literate code (as we preached incessantly in PDP), so that the next reader will have at least some clue as about what you wrote. This just may be sufficient incentive to modify rather than rebuild from scratch.</p> + +<p>Ever the optimist, &mdash;Mitch</p> \ No newline at end of file diff --git a/blog/feeds/all.rss.xml b/blog/feeds/all.rss.xml new file mode 100644 index 00000000..9501b212 --- /dev/null +++ b/blog/feeds/all.rss.xml @@ -0,0 +1,3605 @@ + + + + PRL Blog: PRL Blog + PRL Blog: PRL Blog + http://prl.ccs.neu.edu/blog/index.html + Wed, 16 Feb 2022 00:00:42 UT + Wed, 16 Feb 2022 00:00:42 UT + 1800 + + [Ř Overview I (cross-post)](https://www.o1o.ch/lab/entry/1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc.html) + http://prl.ccs.neu.edu/blog/2022/02/16/-r%CC%8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-02-16-r-CC-8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html + Wed, 16 Feb 2022 00:00:42 UT + Olivier Flückiger + + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + http://prl.ccs.neu.edu/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + Thu, 06 Jan 2022 17:56:08 UT + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> + + Deep and Shallow Types + http://prl.ccs.neu.edu/blog/2020/12/23/deep-and-shallow-types/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-12-23-deep-and-shallow-types + Wed, 23 Dec 2020 18:21:55 UT + Ben Greenman + +<p>I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:</p> + +<ul> + <li><a href="http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020</a></li></ul> + +<p>To the PRL: thanks for a wonderful 6.5 years.</p> +<!-- more--> + +<h3 id="abstract">Abstract</h3> + +<blockquote> + <p>The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound&rsquo;&rsquo; mixed-typed languages from others that enforce type obligations locally rather than globally.</p> + <p>In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.</p> + <p>My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.</p> + <p>I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.</p></blockquote> + +<p>Next year, I&rsquo;ll be a <a href="https://cifellows2020.org">CI Fellow</a> at Brown.</p> + + Transient for Optional and Keyword Functions + http://prl.ccs.neu.edu/blog/2020/11/12/transient-for-optional-and-keyword-functions/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-11-12-transient-for-optional-and-keyword-functions + Thu, 12 Nov 2020 10:15:16 UT + Ben Greenman + +<p>A short adventure into the depths of optional and/or keyword functions in Racket.</p> +<!-- more--> + +<hr /> + +<p>Transient, or rather <em>the Transient semantics for a mixed-typed language</em>, is one way to let statically-typed code safely interact with untyped code. You can read all about it in <a href="http://hdl.handle.net/2022/23172">Michael Vitousek&rsquo;s 2019 dissertation</a> or <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my 2020 dissertation</a>, and you can see how it compares to other mixed-typed semantics <a href="http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">here</a>. The idea is to give up on <a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">behavioral type guarantees</a> and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called <em>shape checks</em>; for example:</p> + +<ul> + <li>if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;</li> + <li>if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and</li> + <li>if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.</li></ul> + +<p>Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type <strong>T</strong> into a shape check, and we have a function with type <strong>T</strong> that needs to check its inputs. The question is how to actually do the check in Racket v7.9.</p> + +<p>In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if <strong>T = (-&gt; Symbol Boolean)</strong> then we need to check the shape <strong>symbol?</strong> of the domain type <strong>Symbol</strong>:</p> + +<pre><code>;; source code +(: f (-&gt; Symbol Boolean)) +(define (f sym) + (eq? sym 'hola)) + +;; ===&gt; + +;; imaginary (but realistic) rewritten code +(define (f sym) + (assert sym symbol?) + (eq? sym 'hola))</code></pre> + +<p>A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type <strong>T</strong> accepts 1 to 3 inputs:</p> + +<pre><code>;; source code +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) +(define (g #:a a [b 'b] #:c [c #f]) + (if a b (if c 'left 'right))) + +;; ===&gt; + +;; imaginary, unrealistic rewritten code +(define (g #:a a [b 'b] #:c [c #f]) + (assert a boolean?) + (assert b symbol?) + (assert c void?) + (if a b (if c 'left 'right)))</code></pre> + +<p>Good &mdash; we basically know what we want. If the Racket core language had optional and keyword functions, then we&rsquo;d be done.</p> + +<p>But no, Racket expands these optional/keyword functions into primitive <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._lam))"><strong>lambda</strong></a> and <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._case-lam))"><strong>case-lambda</strong></a> forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.</p> + +<p>Let&rsquo;s keep digging.</p> + +<p>From now on, &ldquo;Shallow&rdquo; or &ldquo;Shallow TR&rdquo; refers to my implementation of Transient for Typed Racket (TR). We&rsquo;ll talk about Shallow instead of &ldquo;Transient&rdquo; in case future work reveals a better way to implement the Transient idea.</p> + +<h2 id="false-start-follow-the-type">False Start: Follow the Type</h2> + +<p>Beware &mdash; Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function <strong>g</strong> above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional <strong>#:c</strong> keyword lead to a shape-check failure because the variable <strong>c</strong> gets the default value <strong>#f</strong> instead of a void value. Concretely, the third assert from above fails:</p> + +<pre><code>(define (g #:a a [b 'b] #:c [c #f]) + .... + (assert c void?) ;; fails if c is the #f default value + ....)</code></pre> + +<p>The problem arises from subtyping. According to the annotations, the function <strong>g</strong> has an external type that is less precise than the internal type that validates the function body:</p> + +<pre><code>;; external type T +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) + +;; internal type T2, subtype of external (T2 &lt;: T), validates body +(: g (-&gt;* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))</code></pre> + +<p>Thanks to this external / internal distinction, the following easy rewrite idea, <em>Solution 0</em>, fails. Despite the failure, this first solution is a useful starting point for a success.</p> + +<h4 id="solution-0-step-1-mimic-the-typechecker">Solution 0, Step 1: Mimic the Typechecker</h4> + +<p>Shallow TR uses the same type checker as classic <em>Deep</em> TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.</p> + +<p>Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.</p> + +<p>Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (<a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-expr-unit.rkt#L274-L295">typecheck/tc-expr-unit.rkt</a>). The omitted code (<strong>&hellip;.</strong>) does actual type checking:</p> + +<pre><code>(define (tc-expr/check/internal form expected-type) + .... + (syntax-parse form + #:literal-sets (kernel-literals tc-expr-literals) + .... + [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^) + ....] + [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>Ok! Those two patterns say a lot about the expansion of optional and keyword functions:</p> + +<ol> + <li>Both forms expand to a <strong>let-values</strong> that binds one function <strong>fun</strong>.</li> + <li>TR uses the syntax classes <strong>kw-lambda^</strong> and <strong>opt-lambda^</strong> to tell these particular <strong>let-values</strong> apart from others.</li></ol> + +<p>Shallow TR can use exactly these patterns to recognize optional/keyword functions.</p> + +<h4 id="solution-0-step-2-parse-the-domain-type">Solution 0, Step 2: Parse the Domain Type</h4> + +<p>Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function&rsquo;s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:</p> + +<pre><code> [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/type-table.rkt#L80"><strong>type-of</strong></a> function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, <strong>fun</strong>, has a type. Great.</p> + +<p>Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:</p> + +<ul> + <li>The type of <strong>fun</strong> could be a straightforward <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L693"><strong>Fun type</strong></a>, but it could also be a: <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L701"><strong>DepFun type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L521"><strong>Poly type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L531"><strong>PolyDots type</strong></a>, or even a <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L900"><strong>Union type</strong></a>.</li> + <li>Each part of the domain type corresponds to one parameter of the <strong>fun</strong> expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in <strong>fun</strong>, or the mandatory keywords?</li></ul> + +<h4 id="solution-0-step-3-insert-a-shape-check">Solution 0, Step 3: Insert a Shape Check</h4> + +<p>Once we have the target <strong>fun</strong> expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse <strong>fun</strong> to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting <strong>fun</strong> goes something like this:</p> + +<pre><code>(syntax-parse fun + [(#%plain-lambda formals . body) + #:with (shape-check ...) + (make-shape-checks #'formals (type-of fun)) + #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])</code></pre> + +<p>The rewritten function executes shape checks immediately, and then proceeds with the <strong>body</strong> after validating each actual parameter.</p> + +<h2 id="on-the-trail-optkey-expansion">On the Trail: optkey Expansion</h2> + +<p>Our <em>Solution 0</em> fails because the type of the <strong>fun</strong> expression that it gets from the type-checked code is an external type. In terms of the <strong>g</strong> function from above, <em>Solution 0</em> uses the type <strong>Void</strong> instead of the internal type <strong>(U Void #f)</strong> to check the <strong>c</strong> parameter. To get internal types, we need to look closer at <strong>fun</strong> and the rest of the optional/keyword expansion.</p> + +<p>Let&rsquo;s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.</p> + +<p>If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function <strong>X</strong> create a module <strong>test.rkt</strong> like this:</p> + +<pre><code>#lang racket/base + +(define _ignore + (let () + X + (void)))</code></pre> + +<p>Invoke the expander with <code>raco expand test.rkt &gt; test.rkt.txt</code> and explore the generated <strong>.txt</strong> file.</p> + +<h3 id="example-1-mandatory-keyword">Example 1: mandatory keyword</h3> + +<p>The source is a function with one mandatory positional argument and one optional positional argument.</p> + +<pre><code>(lambda (x [y 0]) + (+ x y))</code></pre> + +<p>Expansion generates a <strong>case-lambda</strong> that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function <strong>F</strong> that expects two arguments, resolves defaults in a different way, and executes the function body.</p> + +<pre><code>(let-values (((F) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F x '0)) + ((x y1) (#%app F x y1))))</code></pre> + +<p>Note: the expression <strong>(if &rsquo;#f &rsquo;0 y1)</strong> in the generated <strong>F</strong> function is equal to <strong>y1</strong> alone. In general, the <strong>if</strong> is for default expressions. (<a href="https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/">Unlike Python</a>, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a <strong>#f</strong> test. A general-purpose optimizer can remove this test before the code runs.</p> + +<h3 id="example-2">Example 2:</h3> + +<p>The source is a function with one mandatory positional argument and one mandatory keyword argument:</p> + +<pre><code>(lambda (x #:y y) + (+ x y))</code></pre> + +<p>Expansion generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> expects a list of keywords, a list of arguments, and a final argument. The purpose of <strong>F1</strong> is to organize a call to <strong>F0</strong>.</li> + <li><strong>lifted/2</strong> is the constructor for a generated struct type. Other functions help the struct call <strong>F1</strong>. Nevermind the details; I don&rsquo;t fully understand them either.</li></ul> + +<p>The important piece for Shallow TR is the <strong>F0</strong> function because the goal of rewriting is to protect the original function body against untyped inputs.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) y1)) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y1) (#%app car given-args))) + (#%app F0 y1 x3))))) + (#%app + lifted/2 + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l2571) given-kws)) + (if (#%app pair? l2571) + (if (#%app eq? (#%app car l2571) '#:y) + (#%app null? (#%app cdr l2571)) + '#f) + '#f)) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + '(#:y) + '(#:y))))</code></pre> + +<h3 id="example-3">Example 3:</h3> + +<p>The source is a function with one mandatory positional argument and one optional keyword argument:</p> + +<pre><code>(lambda (x #:y [y 0]) + (+ x y))</code></pre> + +<p>Expansion again generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments, resolves the optional default, and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> calls <strong>F0</strong></li> + <li>At the bottom, there are two <strong>case-lambda</strong> functions that call <strong>F1</strong></li></ul> + +<p>Again, the <strong>F0</strong> function is the focal point for Shallow TR rewriting.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y2) (#%app pair? given-kws))) + (let-values (((y1) + (if y2 (#%app car given-args) '0))) + (#%app F0 y1 x3)))))) + (#%app + make-optional-keyword-procedure + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l1571) given-kws)) + (let-values (((l1571) + (if (#%app null? l1571) + l1571 + (if (#%app eq? (#%app car l1571) '#:y) + (#%app cdr l1571) + l1571)))) + (#%app null? l1571))) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + null + '(#:y) + (case-lambda + ((x) (#%app F1 null null x))))))</code></pre> + +<h2 id="solution-the-shallow-tr-rewrite-strategy">Solution: The Shallow TR Rewrite Strategy</h2> + +<p>All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a <strong>let-values</strong> form:</p> + +<pre><code>(let-values (((f) fun)) . body)</code></pre> + +<p>Furthermore, the generated <strong>fun</strong> is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the <strong>fun</strong> from <em>Example 3</em> again; it has formal parameters for the keyword arg. and the mandatory arg., and one <strong>let-values</strong> to resolve each parameter:</p> + +<pre><code> (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y)))))</code></pre> + +<p>Another experiment with <strong>type-of</strong> shows that the right-hand side of each <strong>let-values</strong> has an internal type annotation. Excellent! Both <strong>(type-of x3)</strong> and <strong>(type-of (if &rsquo;#f &rsquo;0 y1))</strong> are the right types for shape checks. Shallow TR can:</p> + +<ul> + <li>inspect the <strong>let-values</strong> one-by-one;</li> + <li>convert the type of each right-hand expression to a shape predicate; and</li> + <li>rewrite each right-hand <strong>expr</strong> into <strong>(assert expr shape?)</strong>.</li></ul> + +<p>This should work! In fact, we can do slightly better:</p> + +<ul> + <li>when the right-hand expression is a conditional <strong>(if test default-expr supplied-arg)</strong></li> + <li>then Shallow only needs to check the supplied arg: <strong>(if test default-expr (assert supplied-arg shape?))</strong></li></ul> + +<p>Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.</p> + +<h2 id="a-problem-with-methods-and-a-bugfix">A Problem with Methods and a Bugfix</h2> + +<p>Currently, Shallow TR rewrites optional and keyword functions using the <strong>let-values</strong> plan described above. Each formal parameter has one <strong>let-values</strong> binding, and the type on each bound expression defines the shape check.</p> + +<p>Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we <a href="https://github.com/racket/racket/pull/3182">fixed the class expander</a>.</p> + +<p>First, here is a class with one method that runs correctly. The method <strong>f</strong> accepts an optional positional argument <strong>x</strong>; the default value of <strong>x</strong> is the current value of the field <strong>my-num</strong> (fields are mutable):</p> + +<pre><code>(define c0% + (class object% + (super-new) + (field (my-num 2)) + (define/public (f [x my-num]) + (+ x x))))</code></pre> + +<p>Second, here is a similar method that fails. This time, the default is an immediate value <strong>2</strong>:</p> + +<pre><code>(define c1% + (class object% + (super-new) + (define/public (f [x 2]) + (+ x x))))</code></pre> + +<p>Running a call <strong>(send o1 f)</strong> used to raise a shape-check failure about a strange value:</p> + +<blockquote> + <p>shape error: Expected a real number, got <code>#&lt;unsafe-undefined&gt;</code></p></blockquote> + +<p>What is going on?</p> + +<p>It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:</p> + +<pre><code>(lambda (x [y z]) + (+ x y))</code></pre> + +<p>Expansion generates a function <strong>F0</strong> that checks for the undefined value, and an outer <strong>case-lambda</strong> that supplies undefined when the default is needed:</p> + +<pre><code>(let-values (((F0) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) + (if (#%app eq? y1 unsafe-undefined) + z + y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F0 x unsafe-undefined)) + ((x y1) (#%app F0 x y1))))</code></pre> + +<p>That&rsquo;s the normal way that <strong>unsafe-undefined</strong> shows up: the <a href="https://github.com/racket/racket/blob/c0ff11e27bd28e070c20b7a9b0f7365f8f2b665a/racket/collects/racket/private/kw.rkt">expander for optional/keyword functions</a> looks for default expressions vs. default values and uses the undefined value for expressions.</p> + +<p>Three other facts conspired to make the problem with optional methods:</p> + +<ol> + <li>Typed Racket also looks for default expressions vs. default values (search for <strong>immediate-default</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/base-env/annotate-classes.rkt">here</a>). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the <strong>unsafe-undefined</strong> value (search for <strong>-Unsafe-Undefined</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/kw-types.rkt">here (kw)</a> and <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-lambda-unit.rkt">here (opt)</a>).</li> + <li>The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.</li> + <li>Shallow TR pushes default expression checks <strong>(if test default-expr supplied-arg)</strong> to the <strong>supplied-arg</strong> instead of wrapping the whole <strong>if</strong> form.</li></ol> + +<p>In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound&mdash;but harmless because the false assumption was guarded by an <strong>if</strong> test for <strong>unsafe-undefined</strong>. Running Shallow TR revealed the unsoundness with its eager shape check.</p> + +<p>Again, the resolution was to fix the class expander (<a href="https://github.com/racket/racket/pull/3182">racket/racket #3182</a>). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.</p> + +<h2 id="lessons">Lessons</h2> + +<ol> + <li>Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.</li> + <li>Digging into the expansion is sometimes necessary. There are at least three places that do so&mdash;the class expander, TR, and Shallow TR&mdash;and unfortunately they all need to cooperate.</li> + <li>The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my dissertation</a> lists them all.</li></ol> + + Transient Answers Old Questions + http://prl.ccs.neu.edu/blog/2020/10/15/transient-answers-old-questions/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-10-15-transient-answers-old-questions + Thu, 15 Oct 2020 13:32:12 UT + Ben Greenman + +<p>Several old questions from the Typed Racket mailing list have new and simple answers under a &ldquo;transient&rdquo; Typed Racket.</p> +<!-- more--> + +<hr /> + +<p>For the past few months, I&rsquo;ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the <a href="https://github.com/racket/typed-racket/pull/952">RFC</a> and <a href="https://github.com/racket/typed-racket/pull/948">pull request</a>.</p> + +<p>The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that&rsquo;s about it. By contrast, types are much stronger in classic Typed Racket.</p> + +<p>Shallow Racket&rsquo;s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the <a href="https://groups.google.com/g/racket-users">Racket mailing list</a>. There were a surprising number of questions that went like this:</p> + +<blockquote> + <p><strong>Q.</strong> Hey, I ran a program expecting <em>X</em> to happen, but <em>Y</em> happened instead. Is this a bug?</p> + <p><strong>A.</strong> No, Typed Racket has to do <em>Y</em> because of its strong types.</p></blockquote> + +<p>&hellip; but changing to shallow types gives the <em>X</em> behavior! Here are their stories.</p> + +<p>Going forward, <strong>Deep</strong> refers to normal Typed Racket and <strong>Shallow</strong> refers to Shallow Typed Racket.</p> + +<hr /> + +<h2 id="higher-order-value-as-any">Higher-Order Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ">groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ</a></p> + +<h4 id="on-20180416-mailoo-wrote">On 2018&ndash;04&ndash;16, <em>mailoo</em> wrote:</h4> + +<blockquote> + <p> I play a little with the &ldquo;Any&rdquo; type (due to &lsquo;dynamic-require&rsquo; which return Any), and I&rsquo;m not able to cast them back in a function.</p> + <p> I (over) simplify my question with this little program :</p></blockquote> + +<pre><code>(: p Any) +(define (p i) (displayln i)) + +; Here I want to get back my function +(define proc (cast p (-&gt; Integer Void))) +(proc 2) </code></pre> + +<blockquote> + <p> but I get this error when I try to execute the function :</p></blockquote> + +<pre><code>; contract violation +; Attempted to use a higher-order value passed as `Any` in untyped code: #&lt;procedure:p&gt; </code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> raises an error because it must enforce the <code>Any</code> type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> prints 2 and returns void. No error. Same goes for dynamic-require.</p> + +<hr /> + +<h2 id="parametric-contract-affects-untyped-code">Parametric Contract Affects Untyped Code</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ">groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ</a></p> + +<h4 id="on-20191215-john-clements-wrote">On 2019&ndash;12&ndash;15, John Clements wrote:</h4> + +<blockquote> + <p> It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:</p></blockquote> + +<pre><code> #lang typed/racket + + (require/typed racket/list + [index-of (All (T) ((Listof T) T -&gt; (U False Natural)))]) + + (index-of '(n s e w) 'n) ;; returns... #f? </code></pre> + +<blockquote> + <p> In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).</p> + <p> I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces parametricity for <code>All</code> types, and this throws off the equality function that index-of uses internally.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> returns 0.</p> + +<p>ps John, thanks very much for working on <a href="https://adventofcode.com">Advent of Code</a> and mailing the list!</p> + +<hr /> + +<h2 id="unable-to-protect-opaque-value-as-any">Unable to Protect Opaque Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ">groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ</a></p> + +<h4 id="on-20191211-marc-kaufmann-wrote">On 2019&ndash;12&ndash;11, Marc Kaufmann wrote:</h4> + +<blockquote> + <p>I have one file called <code>type-test.rkt</code> with the following</p></blockquote> + +<pre><code>#lang typed/racket + +(require (only-in typed/web-server/http response/xexpr response)) + +(provide f2) + +(: f2 (-&gt; (U response Any))) +(define (f2) + (define x '(body (h1 "Try it"))) + (: resp response) + (define resp (response/xexpr x)) + resp)</code></pre> + +<blockquote> + <p>Then I have another <em>untyped</em> file for a servlet:</p></blockquote> + +<pre><code>#lang racket + +(require "type-test.rkt" + web-server/servlet + web-server/servlet-env) + +(define (start req) + (f2)) + +(serve/servlet start + #:servlet-regexp #rx"" + #:launch-browser? #false + #:port 8080)</code></pre> + +<blockquote> + <p>Notice that I am telling [f2] that <code>resp</code> is of type <code>response</code>. Yet, when I run the server with <code>start</code> [&hellip;.] I get the following result:</p> + <p>(f2): Error, see below.</p> + <p>The error is:</p></blockquote> + +<pre><code>f2: broke its own contract + any-wrap/c: Unable to protect opaque value passed as `Any` + value: #&lt;response&gt; + in: the range of + (-&gt; Any)</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the <code>Any</code> type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.</p> + +<hr /> + +<h2 id="type-inference-installs-a-precise-type">Type Inference Installs a Precise Type</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ">groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ</a></p> + +<h4 id="on-20200214-john-clements-wrote">On 2020&ndash;02&ndash;14, John Clements wrote:</h4> + +<blockquote> + <p>I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.</p> + <p>Here’s a program:</p></blockquote> + +<pre><code>#lang typed/racket + +(define-type Store (Mutable-HashTable Integer Value)) +(define-type Value (U Real Boolean String)) + +(define top-store + (cast + (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f))) + Store)) + +(hash-set! top-store 5 1234)</code></pre> + +<blockquote> + <p>It fails with this error:</p></blockquote> + +<pre><code>contract violation +expected: (or/c (and/c byte? positive?) #t #f) +given: 1234 +in: the values of +the 3rd conjunct of +(and/c hash? + hash-mutable? + (hash/c exact-integer? + (or/c (and/c byte? positive?) #t #f) + #:immutable #f))</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p>First off, <strong>Deep</strong> runs fine after swapping <code>cast</code> for <code>ann</code>.</p> + +<p>Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.</p> + +<p>The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.</p> + +<hr /> + +<h2 id="same-arity-functions-in-a-case-lambda">Same-Arity Functions in a Case Lambda</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ">groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ</a></p> + +<h4 id="on-20190705-ryan-kramer-wrote">On 2019&ndash;07&ndash;05, Ryan Kramer wrote:</h4> + +<blockquote> + <p>In the code below, can <code>maybe-car</code> have the given type [&hellip;.]?</p></blockquote> + +<pre><code>#lang typed/racket + +(module untyped racket + (provide maybe-car) + (define (maybe-car x) + (cond + [(pair? x) (car x)] + [else x]))) + +(require/typed + 'untyped + [maybe-car (All (a b) (case-&gt; + (-&gt; (Pairof a b) a) + (-&gt; a a)))])</code></pre> + +<blockquote> + <p>[Current error:]</p></blockquote> + +<pre><code>Type Checker: + Type (All (a b) (case-&gt; (-&gt; (Pairof a b) a) (-&gt; a a))) + could not be converted to a contract: + function type has two cases of arity 1</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the type with a Racket <code>or/c</code> contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don&rsquo;t know which to apply to an incoming function.</p> + +<p>Note, the &ldquo;Type Checker&rdquo; error message is much better than what or/c would give on its own.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> simply checks that maybe-car accepts both arities inside the case-&gt; type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.</p> + +<hr /> + +<h3 id="immutable-type-affects-untyped-code">Immutable Type Affects Untyped Code</h3> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ">groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ</a></p> + +<h4 id="on-20200217-bertrand-augereau-wrote">On 2020&ndash;02&ndash;17, Bertrand Augereau wrote:</h4> + +<blockquote> + <p>Hello everybody, I&rsquo;m trying to gradually type my script to make it a proper app (yes I&rsquo;m a static-ish guy) and I have an issue (Racket 7.6 CS).</p></blockquote> + +<pre><code>; racket_mod.rkt: +#lang racket + +(provide (struct-out s)) +(provide list-of-s) +(provide set-list-of-s!) + +(struct s (a)) +(define list-of-s '()) +(define (set-list-of-s! los) + (set! list-of-s los))</code></pre> + +<pre><code>; racket_mod_typed.rkt: +#lang typed/racket + +(provide (struct-out s2)) +(provide list-of-s2) +(provide set-list-of-s2!) + +(struct s2 ([a : Natural])) +(define list-of-s2 '()) +(define (set-list-of-s2! [los : (Listof s2)]) + (set! list-of-s2 los))</code></pre> + +<pre><code>; racket_main.rkt: +#lang racket + +(require "racket_mod.rkt") +(require "racket_mod_typed.rkt") + +(define los (list (s 1) (s 2))) +(set-list-of-s! los) +(displayln list-of-s) + +(define los2 (list (s2 1) (s2 2))) +(set-list-of-s2! los2) +(displayln list-of-s2)</code></pre> + +<blockquote> + <p>list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces the type of <code>list-of-s2</code> with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.</p> + +<p>Here&rsquo;s a step-by-step breakdown:</p> + +<ol> + <li>the typed module creates an empty list-of-s2</li> + <li>the main module imports the list and receives a new copy</li> + <li>the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable</li> + <li>the main module reads from its copy &mdash; and it&rsquo;s still empty</li></ol> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> lets the original list travel to untyped code. There are no contracts in the way.</p> + +<h2 id="discussion">Discussion</h2> + +<p>Wow! It&rsquo;s great to see that Shallow Racket works &ldquo;as expected&rdquo; on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.</p> + +<p>If you have a similar experience with a deep-types error, let me know.</p> + +<p>Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error &mdash; and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.</p> + +<p>Shallow Typed Racket is coming soon. Follow the <a href="https://github.com/racket/typed-racket/pull/948">pull request</a> or watch the Racket release notes for news.</p> + +<h3 id="links">Links</h3> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">Larger example</a> where Shallow misses an error that Deep catches</li> + <li>Michael M. Vitousek <a href="http://hdl.handle.net/2022/23172">invented</a> the Transient semantics and implemented it in <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>.</li> + <li>My <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-thesis-2020">upcoming dissertation</a> has lots more to say about Shallow Typed Racket.</li></ul> + +<p><em>Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.</em></p> + + The Typed Racket Optimizer vs. Transient + http://prl.ccs.neu.edu/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + Wed, 15 Jan 2020 12:16:35 UT + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> + + PRL Offsite 2019 Retrospective + http://prl.ccs.neu.edu/blog/2019/12/12/prl-offsite-2019-retrospective/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-12-12-prl-offsite-2019-retrospective + Thu, 12 Dec 2019 12:51:53 UT + Ben Greenman, Olek Gierczak + +<p>On November 11th 2019, the PRL had a private offsite meeting at the <a href="https://mtwyouth.org">More Than Words bookstore</a> in downtown Boston. For future offsite organizers, this post records what happened and how.</p> +<!-- more--> + +<h2 id="early-planning-goals">Early Planning, Goals</h2> + +<p>Every Fall, the PRL holds a kickoff meeting to assign <a href="http://prl.ccs.neu.edu/contact">roles</a> for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.</p> + +<blockquote> + <p>Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the <a href="http://www.warrencenter.com">Warren Conference Center</a> is <a href="https://www.northeastern.edu/events/northeastern-owned-off-campus-venues">Northeastern University&rsquo;s only franchised</a> off-campus venue.</p></blockquote> + +<p>The <strong>motivation</strong> for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students &mdash; both to get to know the group, and to become known. They&rsquo;re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">Spring 2017 edition</a> of HOPL.</p> + +<p>Our primary <strong>goal</strong> this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.</p> + +<p>In light of the main goal, we prefer to call the event an &ldquo;offsite&rdquo; (or &ldquo;offsite meeting&rdquo;) rather than a &ldquo;retreat&rdquo; because the target was an informative day rather than a relaxing one.</p> + +<h2 id="booking-and-logistics">Booking and Logistics</h2> + +<p>Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that&rsquo;s all we had.</p> + +<h3 id="date--nov-11">Date = Nov 11</h3> + +<p>In the beginning, we chose Veterans&rsquo; Day (2019&ndash;11&ndash;11) and Northeastern <a href="https://registrar.northeastern.edu/app/uploads/2019-2020-University-Wide-Calendar-List.pdf">reading day</a> (2019&ndash;12&ndash;05) as possible dates. We ended up with Veterans&rsquo; Day.</p> + +<p>A small number of lab members were opposed to Veterans&rsquo; Day. They gave two reasons: the Fall semester is especially busy, and Veterans&rsquo; Day is a federal holiday.</p> + +<h3 id="venue--mtw">Venue = MTW</h3> + +<p>The first venue we tried was the <a href="http://www.warrencenter.com">Warren Conference Center</a> in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the <a href="https://www.khoury.northeastern.edu/people/chelsea-smith/">Khoury Events Team</a> to reach out on our behalf. They returned with an email address and <a href="/blog/static/warren-center-meetings-and-retreats.pdf">event menu</a> (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.</p> +<!-- Donald Pepple (Northeastern) made contact with Christine Barisano (Framingham)--> + +<p>Second, we contacted the <a href="https://alumni.northeastern.edu/about/alumni-center/">Northeastern University Alumni Center</a>. They were not open on Veterans&rsquo; Day 2019.</p> + +<p>Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: <a href="https://mtwyouth.org">More Than Words</a>.</p> + +<p>More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!</p> + +<blockquote> + <p>More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you&rsquo;ll see a mix of &ldquo;employees&rdquo; and &ldquo;volunteers&rdquo; helping out.</p></blockquote> + +<p>Booking was complicated, though, by the fact that Northeastern requires <a href="http://catalog.northeastern.edu/graduate/health-sciences/academic-policies-procedures/liability-insurance/">liability insurance</a> for all off-campus events. If <strong>you</strong> are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.</p> + +<p>Here are more facts about MTW:</p> + +<ul> + <li>the theater space seats 30 with lots of extra space</li> + <li>the two long tables in the reading room can seat about 20</li> + <li>the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor</li> + <li>there&rsquo;s no whiteboard, but MTW can supply an easel stand</li> + <li>much of the furniture in store is antique and for sale, so be careful using it &mdash; both to avoid damaging it, and because antiques can be oddly-shaped</li> + <li>the bookstore was closed on Veterans&rsquo; Day, but we were able to have our event and buy books</li> + <li>MTW may have fridge space to temporarily hold leftovers</li> + <li>closest T station: <a href="https://www.mbta.com/stops/place-tumnl">Tufts Medical Center</a></li></ul> + +<h3 id="keynote--none">Keynote = None</h3> + +<p>The original, ambitious plan was to invite two keynote speakers &mdash; one working in industry and one from academia &mdash; to enrich the offsite with new knowledge. And because this was the PRL&rsquo;s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)</p> + +<p>We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.</p> + +<p>Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).</p> +<!-- to future planners: we invited Kathi Fisler; she had grant-writing plans for that day and would be interested in coming to a future offsite--> + +<h3 id="food-coffee-tea">Food, Coffee, Tea</h3> + +<p><a href="https://flourbakery.com/">Flour Bakery + Cafe</a> provided lunch. For most days, you can order from Flour using an <a href="https://flourbakery.com/menu/catering/catering-information/">online form</a>. For holidays, you may need to send an email &mdash; that&rsquo;s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan &amp; gluten-free <em>everything spiced</em> salads.</p> + +<p><a href="https://www.trycuppacoffee.com/location/">Cuppacoffee</a> provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything &mdash; the order + napkins, milk, cream cheese, butter, and knives &mdash; ourselves because Cuppacoffee is very close to MTW.</p> + +<p><a href="http://www.cmartboston.com/">CMart</a> sold us water and juices. (There is also a Whole Foods near MTW.)</p> + +<p>At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour&rsquo;s <a href="https://youtu.be/kIbhckqanHI">sticky buns</a>.</p> + +<h3 id="event-program">Event Program</h3> + +<p>The following assumptions/goals constrained the schedule for the day:</p> + +<ul> + <li>give all 17 on-campus Ph.D. students a talk slot; talks must <strong>either</strong> communicate one technical idea from recent work <strong>or</strong> say what led the speaker to apply to a PL Ph.D. program</li> + <li>allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)</li> + <li>do not make the audience sit for more than 1 hour at a time</li> + <li>maximize the number and length of breaks (to encourage discussions)</li> + <li>include a relevant and fun welcome activity</li> + <li>save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions</li></ul> + +<p>We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.</p> + +<p>The activity was <em>codewalks</em>. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition/blob/master/src/main/java/com/seriouscompany/business/java/fizzbuzz/packagenamingpackage/impl/math/arithmetics/IntegerDivider.java">IntegerDivider</a> class from a silly FizzBuzz implementation and a solution to the <a href="https://en.wikipedia.org/wiki/Dutch_national_flag_problem">Dutch national flag problem</a>). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and <a href="https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/">egoless programming</a>.</p> + +<p>The talks ran conference-style. One organizer played &ldquo;session chair&rdquo; to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.</p> + +<p>For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.</p> + +<h2 id="looking-back-at-the-details">Looking Back at the Details</h2> + +<p>Reading Day in the Spring may be a good, free date for future retreats.</p> + +<p>We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.</p> + +<p>The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we&rsquo;d keep the morning breaks short &mdash; just enough to use the restroom and grab a coffee &mdash; and aim for 30-min breaks in the afternoon.</p> + +<p>Groups of three 15-minute talks worked well, but groups of four talks might be equally good.</p> + +<p>Perhaps each speaker should get the chance to pick a talk length. <a href="https://nepls.org/">NEPLS</a>, for example, allows a choice between 5-min. and 30-min. talks.</p> + +<p>The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick &ldquo;topics&rdquo; beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.</p> +<!-- to future organizers: the panel began asking faculty for a mistake in their career and wound up with a defensive flavor--> + +<h2 id="looking-back-overall">Looking Back, Overall</h2> + +<p>This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.</p> + +<p>There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time &mdash; keep the same date &amp; program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.</p> + +<p>With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.</p> + + Complete Monitors for Gradual Types + http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + Thu, 31 Oct 2019 21:58:26 UT + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + Four Kinds of Scoping in R + http://prl.ccs.neu.edu/blog/2019/09/10/four-kinds-of-scoping-in-r/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-four-kinds-of-scoping-in-r + Tue, 10 Sep 2019 11:00:00 UT + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">first</a> and <a href="/blog/2019/09/10/scoping-in-r/">second</a> parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.</p> + +<p>In this third and final part of my blog series, I&rsquo;d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.</p> + +<p>This is a &ldquo;bonus&rdquo; blog post, because I&rsquo;m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.</p> +<!-- more--> + +<h2 id="lexical-scope-and-statistical-computation">Lexical scope and statistical computation</h2> + +<p>In <em>Lexical Scope and Statistical Computation</em>,<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">1</a></sup> Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.</p> + +<p>For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:</p> + +<ul> + <li><em>trivial</em>: no free variables allowed</li> + <li><em>static</em>: a free variable takes its value from a set of global variables</li> + <li><em>lexical</em>: a free variable takes the value of the binding that was in effect when the function was defined</li> + <li><em>dynamic</em>: a free variable takes the value of the most recent assignment to that variable</li></ul> + +<p>Note that under this set of definitions, <em>static scoping</em> is a separate scoping rule and not another name for <em>lexical scoping</em>.</p> + +<p>It is possible to simulate each of strategies in R. For fun, we can even construct &ldquo;factories&rdquo; that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)</p> + +<h3 id="template">Template</h3> + +<p>Our examples will follow the template given below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">factory</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="o">&lt;???&gt;</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">factory</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># error, 0, 1, or 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We want to define a <code>factory</code> that takes a function literal and returns a closure that implements the desired scoping rule.</p> + +<p>Our example consists of three definitions of <code>x</code>. On line 5, we assign <code>0</code> to <code>x</code> at the top level. On line 7, we assign <code>1</code> to <code>x</code> inside function <code>h</code>, where we also create the closure. On line 12, we assign <code>2</code> to <code>x</code> inside the function <code>f</code> and right before we call <code>g</code>, which is the closure.</p> + +<p>Finally, we call <code>f</code> and observe the result:</p> + +<ul> + <li>Under trivial scoping, no free variables are allowed, so <code>f()</code> should result in an error.</li> + <li>Under static scoping, free variables may only refer to global variables, so <code>f()</code> should return <code>0</code>.</li> + <li>Under lexical scoping, free variables refer to the variables in scope when the function was defined, so <code>f()</code> should return <code>1</code>.</li> + <li>Under dynamic scoping, free variables take the value from the most recent assignment, so <code>f()</code> should return <code>2</code>.</li></ul> + +<p>We will implement the body of <code>factory</code> in only 3&ndash;5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to <code>factory</code> on line 10.</p> + +<h3 id="trivial-scoping">Trivial scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeTrivial</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">baseenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeTrivial</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># Error in f(0) : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>substitute</code> returns the unevaluated parse tree for <code>fun</code>. In other words, it obtains the literal argument that was passed for <code>fun</code>. This works because of call-by-need semantics in R: function arguments are packaged up into <em>promises</em>. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">2</a></sup> discusses laziness in R in more detail.</p> + +<p>In this example, on line 8, we call <code>factory</code> with <code>function(a) x+a</code> as the argument for the formal parameter <code>fun</code>. Then, we evaluate that parse tree with <code>eval</code>.</p> + +<p>At this point, <code>res</code> is the closure with expression <code>function(a) x+a</code> and a reference to the environment of <code>makeTrivial</code>. On line 3, we change that reference to <code>baseenv()</code>, which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.</p> + +<p>Therefore, variable lookup in the function literal will only search the base environment, so <code>f()</code> results in an error.</p> + +<h3 id="static-scoping">Static scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeStatic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">globalenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeStatic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, on line 3, we update the environment of <code>res</code> to refer to <code>globalenv()</code>, which is the top-level environment where globals are defined.</p> + +<p>Therefore, variable lookup searches the top-level environment, so <code>f()</code> returns <code>0</code>.</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeLexical</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeLexical</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of <code>res</code> to <code>parent.frame()</code>, which is the environment of the function (<code>h</code>) that called the current function (<code>makeLexical</code>). This allows us to simulate lexical scoping, as if the function literal was evaluated inside <code>h</code>, rather than <code>makeLexical</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>h</code>, so <code>f()</code> returns <code>1</code>.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeDynamic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">function</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="nf">res</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> + <span class="p">}</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeDynamic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, we need another level of indirection. <code>makeDynamic</code> returns an anonymous function literal. The anonymous function takes <code>...</code>, which represents an arbitrary list of arguments, and then on line 5 we call <code>res</code> with those exact arguments. Note that we set the environment of <code>res</code> to be the environment of the <em>caller</em> of the anonymous function. Because of the multiple levels of indirection, the caller is <code>f</code>, on line 17.</p> + +<p>On line 12, <code>makeDynamic</code> returns a closure for the anonymous function. <code>h</code> returns that closure when it is called, and assigns it to <code>g</code>. When <code>g</code> is called on line 17, the function literal <code>function(a) x+a</code> is finally evaluated, and its environment is set to the environment of <code>f</code>, the caller of <code>g</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>f</code>, so <code>f()</code> returns <code>2</code>.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the <a href="/blog/2019/09/10/scoping-in-r/">previous post</a>, it&rsquo;s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.</p> + +<p>And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.</p> + +<p><em>I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>A. Goel and J. Vitek. &ldquo;On the Design, Implementation and Use of Laziness in R,&rdquo; in <em>Proceedings of the ACM in Programming Languages (PACMPL)</em>, vol. 3, no. OOPSLA, 2019. To appear. [<a href="http://janvitek.org/pubs/oopsla19a.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">↩</a></p></li></ol></div> + + Scoping in R + http://prl.ccs.neu.edu/blog/2019/09/10/scoping-in-r/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-scoping-in-r + Tue, 10 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">previous post</a> of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: <em>is R lexically or dynamically scoped?</em></p> +<!-- more--> + +<p>Recall the example program from before:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s examine what happens when we run this example. First, we create a mapping for <code>x</code> in the top-level environment. On line 2, we define a function <code>f</code>, which returns the value of some <code>x</code>. On line 3, we define a function <code>g</code>, which creates a new mapping for <code>x</code>, and then calls <code>f</code>. Note that the assignment on line 4 does <em>not</em> update the definition on line 1.</p> + +<p>When <code>f</code> is called, it needs to look up the value of <code>x</code>. In other words, does the reference of <code>x</code> on line 2 refer to the assignment on line 1 or the assignment on line 4? If <code>f</code> returns <code>1</code>, then the behaviour matches lexical scoping. If it returns <code>2</code>, then the behaviour matches dynamic scoping.</p> + +<p>When we run this example, the result is <code>1</code>. This implies that R is lexically scoped.</p> + +<p>But there&rsquo;s more to this story. In the rest of this blog post, I&rsquo;ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.</p> + +<p>The <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">next and final part</a> of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.</p> + +<h2 id="r-is-lexically-scoped-but">R is lexically scoped, but&hellip;</h2> + +<p>In <em>Evaluating the Design of the R Language</em>,<sup><a href="#2019-09-10-scoping-in-r-footnote-1-definition" name="2019-09-10-scoping-in-r-footnote-1-return">1</a></sup> Morandat, Hill, Osvald, and Vitek write:</p> + +<blockquote> + <p>As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.</p></blockquote> + +<p>In other words, R provides many different &ldquo;escape hatches&rdquo;&mdash;ways to bypass lexical scoping. Additionally, even without escape hatches, some of R&rsquo;s functionality can be surprising.</p> + +<h3 id="functions-environments-and-variables-in-r">Functions, environments, and variables in R</h3> + +<p>Before we look at some examples, I think it&rsquo;s useful to briefly discuss some of the core concepts in R that relate to scoping.</p> + +<ul> + <li> + <p><strong>Functions.</strong> R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.</p></li> + <li> + <p><strong>Environments.</strong> An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the &ldquo;enclosing&rdquo; environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.</p></li> + <li> + <p><strong>Variable lookup.</strong> When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.</p></li> + <li> + <p><strong>Variable assignment.</strong> <code>&lt;-</code> is the variable assignment operator in R. The expression <code>x &lt;- 1</code> assigns the value <code>1</code> to the variable <code>x</code> in the current environment. If a mapping for <code>x</code> already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.</p></li></ul> + +<p>From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.<sup><a href="#2019-09-10-scoping-in-r-footnote-2-definition" name="2019-09-10-scoping-in-r-footnote-2-return">2</a></sup></p> + +<p>On the other hand, variable lookup depends on the run-time state of the program&mdash;names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.</p> + +<p>The following examples will make this clear.</p> + +<h3 id="examples">Examples</h3> + +<h4 id="adding-variable-mappings-at-run-time">Adding variable mappings at run time</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it creates a function <code>g</code> that returns <code>x</code>, assigns <code>2</code> to <code>x</code>, and then calls <code>g</code>. When <code>g</code> is called, it looks up <code>x</code>. Since no mapping is found in <code>g</code>&rsquo;s environment, it searches in the enclosing environment (<code>f</code>&rsquo;s), and finds that <code>x</code> has value <code>2</code>. Therefore, <code>g</code> returns <code>2</code>.</p> + +<p>Note that the <code>x</code> on line 3 is resolved only when function <code>g</code> is called, not when it is defined. However, when <code>g</code> is defined, its environment has a reference to <code>f</code>&rsquo;s environment. Therefore, as long as <code>x</code> is defined <em>before</em> <code>g</code> is called, the lookup will always succeed.</p> + +<p>Here&rsquo;s a second example:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>f</code> is a function that branches on its argument, <code>b</code>. If <code>b</code> evaluates to true, then the expression <code>x &lt;- 2</code> is evaluated, and a mapping for <code>x</code> is created in <code>f</code>&rsquo;s environment. Otherwise, no mapping is created.</p> + +<p>When we look up the value of <code>x</code> on line 5, R will first search the function&rsquo;s environment. If <code>b</code> evaluated to true, then R will find a value for <code>x</code>, which is <code>2</code>. Otherwise, R will search in the enclosing environment of <code>f</code>, and find that <code>x</code> is <code>1</code>.</p> + +<p>Both of these examples vaguely resemble dynamic scoping, in that <code>x</code> takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.</p> + +<h4 id="function-lookup">Function lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> <span class="c1"># not an error</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">(</span><span class="m">42</span><span class="p">)</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.</p> + +<p>In this example, we call <code>g</code> with the argument <code>42</code>, which is not a function. Then, in the body of <code>g</code>, we call <code>f(0)</code> on line 3, which requires looking up <code>f</code>. Although there is an <code>f</code> in the environment of <code>g</code>, its value is <code>42</code>, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so <code>f(0)</code> returns <code>0</code>.</p> + +<p>This behaviour exists because <code>c</code> is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.</p> + +<h4 id="super-assignment">Super assignment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="n">x</span> <span class="o">&lt;&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +<span class="n">x</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>&lt;&lt;-</code> is the &ldquo;super assignment&rdquo; operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.</p> + +<p>In the above program, we define <code>x</code> to be <code>0</code> at the top level, and then define the function <code>f</code>. When we call <code>f</code> on line 7, it assigns <code>1</code> to <code>x</code> on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, <code>f</code> returns <code>x</code>, which is looked up from the local environment and has value <code>1</code>. Finally, line 8 looks up <code>x</code> from the top level environment, which has value <code>2</code>.</p> + +<h4 id="evaluating-arbitrary-code">Evaluating arbitrary code</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">eval</span><span class="p">(</span><span class="nf">parse</span><span class="p">(</span><span class="n">text</span> <span class="o">=</span> <span class="n">t</span><span class="p">))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="s">"x &lt;- 0"</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument <code>t</code>, which happens to be the string <code>"x &lt;- 0"</code>. Then, when line 4 executes, the lookup of <code>x</code> returns <code>0</code>.</p> + +<h4 id="simulating-dynamic-scope">Simulating dynamic scope</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.frame</span><span class="p">())</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 3, we perform an explicit variable lookup for <code>x</code>, but we do so in the environment <code>parent.frame()</code>, which refers to the calling function&rsquo;s environment, in this case, <code>g</code>&rsquo;s environment.. Therefore, the lookup returns <code>2</code>.</p> + +<p>Note that R has a similarly named function, <code>parent.env(e)</code> which returns the <em>enclosing environment</em> of the given environment <code>e</code>.</p> + +<h4 id="constructing-an-arbitrary-environment">Constructing an arbitrary environment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">e</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> + <span class="n">e</span><span class="o">$</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">3</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="n">e</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 3</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it constructs a new environment, <code>e</code>, which is initially empty. (By default, its enclosing environment is the current environment, which is <code>f</code>&rsquo;s.) Next, on line 4, it directly adds a mapping to that environment, assigning <code>3</code> to <code>x</code>. Then, on line 5, the lookup is explicitly done in environment <code>e</code>, so <code>f</code> returns <code>3</code>.</p> + +<h4 id="deleting-mappings">Deleting mappings</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">rm</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.env</span><span class="p">(</span><span class="nf">environment</span><span class="p">()))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># Error in f() : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Not only is it possible to dynamically add and modify mappings in R, but it is also possible to <em>delete</em> mappings. This is what line 3 does: it explicitly removes the mapping for <code>x</code> from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when <code>f</code> is called, the lookup of <code>x</code> fails and an error is raised.</p> + +<h4 id="infinite-loop-during-variable-lookup">Infinite loop during variable lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">enva</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="n">envb</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">enva</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">envb</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">envb</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="nf">environment</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="nf">f</span><span class="p">()</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.</p> + +<p>On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: <code>enva</code>&rsquo;s enclosing environment is <code>envb</code>, and <code>envb</code>&rsquo;s enclosing environment is <code>enva</code>.</p> + +<p>On line 5, we define a function with a free variable, <code>x</code>, but on line 6, we set <code>f</code>&rsquo;s environment to be <code>enva</code>. Finally, we call <code>f</code>.</p> + +<p>When the body of <code>f</code> is evaluated, it needs to look up <code>x</code>. Lookup starts in <code>f</code>&rsquo;s environment, which we set to be <code>enva</code>. Since no mapping for <code>x</code> is found, lookup continues in <code>enva</code>&rsquo;s enclosing environment, which is <code>envb</code>. However, <code>envb</code> is also empty, so lookup continues in its enclosing environment, which is <code>enva</code>, and now lookup results in an infinite loop.</p> + +<h3 id="an-intuition-for-scoping-in-r">An intuition for scoping in R</h3> + +<p>Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1"># example 1</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> + +<span class="c1"># example 2</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>It seems that <code>x</code> takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What&rsquo;s missing from our intuition?</p> + +<p>The key insight is that R is <em>function scoped</em>. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <code>&lt;-</code> <em>does not create a new scope</em>, and it is more useful to think of it as a mutation <em>on the current environment</em>. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)</p> + +<p>In a sense, it might be more accurate to say that R <em>environments</em> are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.</p> + +<h2 id="discussion">Discussion</h2> + +<p>All of this might make you a little uncomfortable, and uncertain about R&rsquo;s scoping rules.</p> + +<p>On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply &ldquo;lexical-like&rdquo; behaviour, and the creators have confirmed that lexical scoping was the intent.</p> + +<p>On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment&mdash;but this is not consistent with R&rsquo;s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>This ambiguity shows up in a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-3-definition" name="2019-09-10-scoping-in-r-footnote-3-return">3</a></sup> where the authors write:</p> + +<blockquote> + <p>Furthermore, because variable scoping in R is dynamic and can be modified at the language level [&hellip;] it cannot be trivially guaranteed that <code>x</code> is going to point to the same data structure throughout the entire execution of the loop.</p></blockquote> + +<p>It is true that a variable <code>x</code> may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.</p> + +<p>It is true that variable <em>lookup</em> is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of <em>dynamic scope</em>, then it would be fair to say that R is dynamically scoped.</p> + +<p>But if your definition of <em>dynamic scope</em> is &ldquo;a variable is bound to the most recent assignment during the program&rsquo;s execution,&rdquo; then it is not correct to say R is dynamically scoped.</p> + +<p>I think we have this ambiguity because <em>scope</em> (the places in a program where a variable can be referenced) and <em>variable lookup</em> or <em>name resolution</em> (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.</p> + +<p>Personally, I prefer the definition of <em>scope</em> that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is <em>both</em> lexically and dynamically scoped, or that R is <em>neither</em> lexically and dynamically scoped.</p> + +<p>I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language&mdash;when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.</p> + +<p>Regardless, it is very easy to miscommunicate, so I think it&rsquo;s important to be very clear and make sure you and your audience know what definitions of scoping you&rsquo;re using!</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>This entire adventure started when we were working on a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-4-definition" name="2019-09-10-scoping-in-r-footnote-4-return">4</a></sup> and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!</p> + +<p>This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.</p> + +<p>To summarize, I would say that <em>scope</em> refers to the places in a program where a variable is visible and can be referenced. Under <em>lexical scoping</em>, the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program. Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment during the program&rsquo;s execution.</p> + +<p>I would say that R <em>aims</em> to be lexically scoped&mdash;it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!</p> + +<p>If you want more content about R and scoping, the <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">third and final part</a> of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.</p> + +<p><strong>Edited 2020/02/21:</strong> For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), <a href="http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/">this blog post</a> has some nice examples and diagrams.</p> + +<p><em>I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.</em></p> + +<p><em>If you liked this post, you may also be interested in the following Twitter threads about R: <a href="https://twitter.com/mhyee/status/1063983175163158531">one</a>, <a href="https://twitter.com/mhyee/status/1067818720532316166">two</a> and <a href="https://twitter.com/mhyee/status/1074744049951739905">three</a>.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>F. Morandat, B. Hill, L. Osvald, J. Vitek. &ldquo;Evaluating the Design of the R Language,&rdquo; in <em>Proceedings of the European Conference on Object-Oriented Programming (ECOOP)</em>, 2012. [<a href="https://doi.org/10.1007/978-3-642-31057-7_6">DOI</a>][<a href="http://janvitek.org/pubs/ecoop12.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. &ldquo;Lexical Scope and Statistical Computing&rdquo;, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-2-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-3-definition" class="footnote-definition"> + <p>L. Stadler, A. Welc, C. Humer, and M. Jordan. &ldquo;Optimizing R Language Execution via Aggressive Speculation,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2016. [<a href="https://doi.org/10.1145/2989225.2989236">DOI</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-3-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-4-definition" class="footnote-definition"> + <p>O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. &ldquo;R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2019. To appear. [<a href="http://janvitek.org/pubs/dls19.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-4-return">↩</a></p></li></ol></div> + + Lexical and Dynamic Scope + http://prl.ccs.neu.edu/blog/2019/09/05/lexical-and-dynamic-scope/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + Thu, 05 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> + + [Conversational Concurrency (cross-post)](https://eighty-twenty.org/2018/01/24/conversational-concurrency) + http://prl.ccs.neu.edu/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-05-11-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency + Sat, 11 May 2019 00:03:16 UT + Tony Garnock-Jones + + + Forgetful and Heedful contracts + http://prl.ccs.neu.edu/blog/2019/04/07/forgetful-and-heedful-contracts/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-04-07-forgetful-and-heedful-contracts + Sun, 07 Apr 2019 23:15:11 UT + Ben Greenman + +<p><em>Forgetful</em> and <em>heedful</em> are two methods for space-efficient contracts developed by <a href="http://www.cs.pomona.edu/~michael/">Michael Greenberg</a> in <a href="https://arxiv.org/abs/1410.2813">2014</a>. These methods were born in the shadow of a third method, <em>eidetic</em>, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.</p> +<!-- more--> + +<hr /> + +<p>Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of <a href="https://www2.ccs.neu.edu/racket/pubs/popl11-dfff.pdf">three parties</a>: the API code, the client code, and the contract between them.</p> + +<p>For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:</p> + +<pre><code> #lang racket/base + + (module sort racket + (provide + (contract-out + [quicksort + (-&gt; (vectorof point/c) void?)])) + + (define point/c (vectorof integer?)) + + (define (quicksort points) + ....)) + + (module client racket + (require (submod ".." sort)) + (quicksort '())) + + (require 'client)</code></pre> + +<pre><code>quicksort: contract violation; + expected a vector + given: '() + in: the 1st argument of + (-&gt; (vectorof (vectorof integer?)) void?) + contract from: + (file.rkt sort) + blaming: (file.rkt client) + (assuming the contract is correct)</code></pre> + +<p>That covers the basics. For an extended introduction to contracts, visit <a href="https://docs.racket-lang.org/guide/contracts.html">The Racket Guide</a>.</p> + +<p>The quicksort example and the related figures are from the paper <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts: Fixing a Pathology of Gradual Typing</em></a></p> + +<h3 id="classic-contracts-and-space-efficiency">Classic contracts and &ldquo;Space Efficiency&rdquo;</h3> + +<p>The <code>(vectorof point/c)</code> contract used above describes a possibly-mutable array whose elements match the <code>point/c</code> contract. Since the array can be mutated, this contract has implications for two parties:</p> + +<ol> + <li>the client module must supply a good array, and</li> + <li>the sorting module must not insert a bad element.</li></ol> + +<p>To enforce the second condition, the <code>vectorof</code> contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:</p> + +<pre><code>(quicksort (vector (vector 4 4) + (vector 2 2) + (vector 1 1) + (vector 3 3)))</code></pre> + +<p>After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a <strong>solid</strong> blue bar for the <strong>write</strong> checks against the sort module and a <em>striped</em> blue bar for the <em>read</em> checks against the client.</p> + +<p><img src="/img/vector-chaperone-0.png" alt="A wrapped vector" /></p> + +<p>In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted &mdash; because every <strong>write</strong> and <em>read</em> applies a contract to the element.</p> + +<p><img src="/img/vector-chaperone-1.png" alt="Layers of element wrappers" /></p> + +<p>On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.</p> + +<p>Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and &ldquo;indirecting&rdquo; future writes/reads through wrappers. These space and time costs can add up.</p> + +<blockquote> + <p>&ldquo;on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times&rdquo; &mdash; <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts</em></a></p></blockquote> + +<p>To fix the problem, researchers have been exploring <em>space-efficient</em> implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and <em>eidetic</em>, <em>forgetful</em>, and <em>heedful</em> are his names for three implementations.</p> + +<p>(Although the goal of this post is to promote <em>forgetful</em> and <em>heedful</em>, we will review all three.)</p> + +<h3 id="eidetic-space-efficiency">Eidetic space-efficiency</h3> + +<p>The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a <em>merge</em> operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal &ldquo;wrapping&rdquo; contracts and their size is bounded by the number (and height) of source-code contracts in the program.</p> + +<p>An eidetic contract is an <code>N</code>-ary tree (for <code>N &gt; 0</code>):</p> + +<ul> + <li>each node represents a higher-order contract combinator, such as <code>vectorof</code></li> + <li>the <code>N</code> children of a node represent the different interactions that the value supports</li> + <li>each leaf is a list of non-higher-order, or <em>flat</em>, contracts</li></ul> + +<p>For example, the <code>(vectorof point/c)</code> source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf">Collapsible Contracts</a> paper has an illustration. Each tree node represents a <code>vectorof</code> contract; these nodes have <code>N=2</code> children because vectors support reads and writes.</p> + +<p>A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists &mdash; if an implementation comes with a <code>contract-stronger?</code> predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.</p> + +<p>Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should <code>contract-stronger?</code> try to solve problems that lack polynomial-time solutions?</p> + +<p>Thankfully, there are at least two &ldquo;compromise&rdquo; alternatives.</p> + +<h3 id="forgetful-space-efficiency">Forgetful space-efficiency</h3> +<!-- "no operation relies on e being a T2, skipping the check doesn't risk soundness" p.12--> +<!-- "In forgetful \lambda_H, we offer a simple solution to space inefficient casts: just forget about them" p.11--> +<!-- "Just the same, when accumulating casts on the stack, we throw away all but the last cast" p.11--> +<!-- "forgetful ... skip[s] checks and change[s] blame labels" p.3--> + +<blockquote> + <p>&ldquo;Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 10, bottom of page 23--></blockquote> + +<p>The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:</p> + +<ol> + <li>the client obligations from the old contract, and</li> + <li>the server obligations from the new contract</li></ol> + +<p>If not, proceed as usual &mdash; by wrapping (an unwrapped value) or raising an error. Every value receives at most <strong>one</strong> wrapper; this wrapper changes as the value flows to different clients.</p> + +<p>Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module <code>A</code> exports a function <code>f</code> with contract <code>(-&gt; T1 T2)</code> to module <code>B</code>, and suppose module <code>B</code> shares this function with a few other client modules using different contracts. As <code>f</code> flows to a new client, it keeps the <code>T1</code> domain check and gets a replacement for the <code>T2</code> codomain check.</p> + +<ul> + <li>Keeping <code>T1</code> ensures that the code inside the function (defined by module <code>A</code>) receives input that matches its expectation.</li> + <li>Replacing <code>T2</code> ensures that each new client receives output that it expects.</li></ul> + +<p>Unfortunately, replacing <code>T2</code> also means that clients of module <code>B</code> cannot trust the <code>T2</code> contract. This contract is not checked, and so forgetful contracts <strong>miss</strong> some errors that would be caught by standard contracts. For the same reason, a bug in module <code>B</code> may go undetected by its clients &mdash; even if a later contract reports an issue, the contract system has no memory that <code>B</code> was partly-responsible.</p> + +<p>Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.</p> + +<h3 id="heedful-space-efficiency">Heedful space-efficiency</h3> + +<p>A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set &mdash; or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.</p> + +<p>To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.</p> + +<p>The heedful method is a compromise between forgetful and eidetic.</p> + +<ul> + <li> + <p>Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of <code>contract-stronger?</code> predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.</p></li> + <li> + <p>Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.</p></li></ul> + +<p>For details, see <a href="https://arxiv.org/abs/1410.2813">the extended version</a> of Michael&rsquo;s POPL 2015 paper. Don&rsquo;t bother searching <a href="http://www.cs.pomona.edu/~michael/papers/popl2015_space.pdf">the conference version</a> &mdash; aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.</p> + +<p><code>*</code> If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to <a href="http://mballantyne.net/">Michael Ballantyne</a> for explaining this to me.</p> + +<h3 id="priorities-and-appearances">Priorities and Appearances</h3> + +<p>The extended version of <em>Space-Efficient Manifest Contracts</em> introduces the forgetful and heedful methods with extreme modesty. It&rsquo;s tempting to skip past them and focus on the eidetic method.</p> + +<blockquote> + <p>&ldquo;Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [<a href="https://dl.acm.org/citation.cfm?doid=1706299.1706342">Siek and Wadler 2010</a>]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen&mdash;warm up exercises for eidetic.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 1, bottom of page 2--></blockquote> + +<p>And yet, at least two other research papers rely on these &ldquo;strawmen&rdquo; &mdash; or rather, the ideas behind the names.</p> + +<p><a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a>, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.</p> + +<blockquote> + <p>&ldquo;if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- page 21--></blockquote> + +<p>This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.</p> + +<blockquote> + <p>&ldquo;removing these casts preserves the soundness of the evaluation while reducing the number of them&rdquo;</p> + <p>&ldquo;while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- p.27--><!-- page 21--></blockquote> +<!-- The followup at POPL 2019 is not forgetful.--> +<!-- It's similar to eager coercions ... keep all types around and error--> +<!-- if there's a new type that doesn't match the old ones.--> +<!-- Also, that paper chooses not to let functions have intersection types,--> +<!-- which kind-of-avoids the questions ... but really the eagerness is key.--> + +<p><a href="https://dl.acm.org/citation.cfm?id=3009849"><em>Big Types in Little Runtime</em></a>, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their <em>transient</em> semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.</p> + +<p>That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value&rsquo;s set. When a mismatch occurs, the semantics <a href="https://www.ccs.neu.edu/home/types/resources/notes/transient-undefined-blame-extract.pdf">tries to report</a> every type in the set that relates to the mismatch.</p> + +<p>And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">ICFP 2018</a>. Where will we see them next?</p> + + PLISS: Learn About PL Implementation in a Castle + http://prl.ccs.neu.edu/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + Sat, 09 Mar 2019 14:40:16 UT + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> + + Writing a paper with Scribble + http://prl.ccs.neu.edu/blog/2019/02/17/writing-a-paper-with-scribble/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-02-17-writing-a-paper-with-scribble + Sun, 17 Feb 2019 16:20:50 UT + Ben Greenman + +<p>This post explains how to get started using Scribble to write a research paper.</p> +<!-- more--> + +<hr /> + +<blockquote> + <p>This post was written using <a href="http://download.racket-lang.org/all-versions.html">Racket 7.1</a> and <a href="https://github.com/racket/scribble/releases/tag/v7.1">Scribble 1.29</a></p></blockquote> + +<p>Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:</p> + +<ul> + <li>the paper can import definitions from the research, keeping a single point of control;</li> + <li>the language&rsquo;s functional abstractions can help manage the writing;</li> + <li>the language&rsquo;s drawing and/or plotting libraries can replace <a href="https://ctan.org/pkg/pgf?lang=en">TikZ</a>;</li> + <li>and you can write unit tests to validate the claims made in the paper.</li></ul> + +<p>Scribble, <a href="http://docs.racket-lang.org/scribble/index.html">the Racket documentation tool</a>, comes with a to-LaTeX compiler and a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> library tailored to the new <a href="https://ctan.org/pkg/acmart?lang=en">ACM paper format</a>. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short &ldquo;getting started&rdquo; guide and links to some larger examples.</p> + +<blockquote> + <p>For a Scribble tutorial, see the links in: <a href="/blog/2017/05/23/building-a-website-with-scribble/index.html">Building a Website with Scribble</a></p></blockquote> + +<h2 id="getting-started-with-">Getting started with <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a></h2> + +<p>The first line of a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> document sets the formatting options (similar to a LaTeX file using <code>acmart.cls</code>). For example, the <a href="https://conf.researchr.org/track/gpce-2018/gpce-2018#Call-for-Papers">GPCE 2018 call for papers</a> asks for anonymized <code>sigplan</code>-format submissions with line numbers and 10 point font. The proper Scribble incantation is:</p> + +<pre><code>#lang scribble/acmart @sigplan @anonymous @review @10pt</code></pre> + +<p>Next, you may want to import some definitions. If we have a file <code>references.rkt</code> (see below for a definition), we can import it as follows:</p> + +<pre><code>@require{references.rkt}</code></pre> + +<p>The third main ingredient is the title and author information:</p> + +<pre><code>@(define neu (affiliation #:institution "Northeastern University")) +@(define anon (email "anon@anon.net")) + +@title{Writing a paper with Scribble} +@author[#:affiliation neu #:email anon]{Ben Greenman} + +@; optional: set the author names in the page headers +@elem[#:style "Sshortauthors"]{B. Greenman}</code></pre> + +<p>The paper is now ready to be written. You can forge ahead with a new <a href="http://docs.racket-lang.org/scribble/base.html#%28def._%28%28lib._scribble%2Fbase..rkt%29._section%29%29">section</a> and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and <a href="http://docs.racket-lang.org/scribble/base.html#%28form._%28%28lib._scribble%2Fbase..rkt%29._include-section%29%29">import</a> content from other modules:</p> + +<pre><code>@include-abstract{abstract.scrbl} +@include-section{introduction.scrbl}</code></pre> + +<p>Finally, the main page is a good place to <a href="https://docs.racket-lang.org/scriblib/autobib.html">generate the bibliography</a>. Assuming this document imports a file like the <code>references.rkt</code> below, this expression inserts a bibliography titled &ldquo;References&rdquo;:</p> + +<pre><code>@generate-bibliography[#:sec-title "References"]</code></pre> + +<p>To build the document, invoke <code>scribble</code> on the command-line with the <code>--pdf</code> or <code>--latex</code> options:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl</code></pre> + +<p>If all goes well, this command generates a <code>FILE.pdf</code> with properly-linked cross references.</p> + +<h3 id="auxiliary-files">Auxiliary Files</h3> + +<p>If you save the code above to a file <code>example.scrbl</code> and save the files below in the same directory, then you should be able to build an <code>example.pdf</code>.</p> + +<p>These files are available in a slightly different format at this link:</p> + +<ul> + <li><a href="https://gitlab.com/bengreenman/scribble-acmart-example">https://gitlab.com/bengreenman/scribble-acmart-example</a></li></ul> + +<h4 id="referencesrkt"><code>references.rkt</code></h4> + +<pre><code>#lang racket/base + +(provide + ~cite citet generate-bibliography + fbf-icfp-2009) + +(require + scriblib/autobib) + +(define-cite ~cite citet generate-bibliography + #:style author+date-square-bracket-style) + +(define icfp "ICFP") + +(define fbf-icfp-2009 + (make-bib + #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools" + #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler") + #:location (proceedings-location icfp #:pages '(109 120)) + #:date 2017))</code></pre> + +<h4 id="abstractscrbl"><code>abstract.scrbl</code></h4> + +<pre><code>#lang scribble/acmart + +A simple Scribble document.</code></pre> + +<h4 id="introductionscrbl"><code>introduction.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require{references.rkt} + +@; start with `title` instead of `section`, because importing via +@; `include-section` shifts all title/section/subsections down one level +@title{Introduction} + +Scribble creates a connection between a stand-alone document and the artifact +it describes@~cite[fbf-icfp-2009].</code></pre> + +<h3 id="q-how-to-debug-scribble-error-messages">Q. How to debug Scribble error messages?</h3> + +<p>If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.</p> + +<p>As a compile-time example, adding <code>@ foo</code> to a document produces the message <code>unexpected whitespace after @</code> and you can either delete the whitespace or change the <code>@</code> to <code>@"@"</code> for a literal <code>@</code>-sign.</p> + +<p>As a run-time example, adding <code>@(+ 2 2)</code> produces this message:</p> + +<pre><code>not valid in document body (need a pre-part for decode) in: 4</code></pre> + +<p>One fix is to convert <code>4</code> to a string, as in <code>@~a[(+ 2 2)]</code>.</p> + +<p>But if something goes wrong when Scribble renders a generated document to PDF, the default error output is <strong>not</strong> likely to help. For example, adding <code>@elem[#:style "oops"]</code> to a document produces a giant message:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl +[[ ... 84K of output ... ]] +Output written on example.pdf (1 page, 277876 bytes). +PDF statistics: + 53 PDF objects out of 1000 (max. 8388607) + 37 compressed objects within 1 object stream + 7 named destinations out of 1000 (max. 500000) + 36877 words of extra memory for PDF output out of 42996 (max. 10000000) + +run-pdflatex: got error exit code + context...: + [[ ... 17 more lines ... ]]</code></pre> + +<p>The best way to debug these messages is to <strong>ignore them</strong> and use a LaTeX compiler directly. For the &ldquo;oops&rdquo; mistake, LaTeX stops at the undefined control sequence &mdash; giving a hint about how to find the problem:</p> + +<pre><code>$ raco scribble --latex FILE.scrbl +$ pdflatex FILE.tex +[[ ... 12KB of output ... ]] +! Undefined control sequence. +l.549 \oops + {} +? </code></pre> + +<h3 id="q-how-to-add-a-latex-style-file">Q. How to add a LaTeX style file?</h3> + +<p>To add extra LaTeX code to the final document, create a new file and include it with the <code>++style</code> command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).</p> + +<pre><code>$ raco scribble ++style style.tex --pdf FILE.scrbl</code></pre> + +<p>Here is an example style file.</p> + +<h4 id="styletex"><code>style.tex</code></h4> + +<pre><code>\settopmatter{printfolios=true,printccs=true,printacmref=true} +% add page numbers etc. + +\overfullrule=1mm +% draw a black rectangle near lines that overflow the margin</code></pre> + +<p>Another way to add extra LaTeX code is to add a <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Flatex-properties..rkt%29._tex-addition%29%29"><code>tex-addition</code></a> style property to the main title. This second approach makes it easy to include more than one file:</p> + +<pre><code>#lang scribble/acmart + +@require[ + (only-in scribble/core make-style) + (only-in scribble/latex-properties make-tex-addition)] + +@(define extra-style-files + (list (make-tex-addition "style.tex"))) + +@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble} + +@; ....</code></pre> + +<h3 id="q-how-to-make-a-figure">Q. How to make a figure?</h3> + +<p>Use the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library to add figures to a document.</p> + +<pre><code>@require[pict scriblib/figure] +@figure[ + "fig:fish" @; figure tag, see `figure-ref` + @elem{A Standard Fish} @; figure caption, appears below the content + @elem{fish = @(standard-fish 90 40)}] @; content</code></pre> + +<p>The content of a figure can be almost anything that would work in the toplevel of the document.</p> + +<h3 id="q-how-to-include-extra-files-pictures-latex">Q. How to include extra files (pictures, LaTeX)?</h3> + +<p>The <code>++extra</code> command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.</p> + +<p>For example, if a document includes the content of an external LaTeX file:</p> + +<pre><code>@elem[#:style "input"]{inline-this.tex}</code></pre> + +<p>then make sure to build the document with a command like this:</p> + +<pre><code>$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl</code></pre> + +<h4 id="inline-thistex"><code>inline-this.tex</code></h4> + +<pre><code>% Raw LaTeX allowed here +$\lambda x.\, x$</code></pre> + +<h3 id="q-what-about-in-line-latex">Q. What about in-line LaTeX?</h3> + +<p>An <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Fcore..rkt%29._element%29%29">element</a> with the <a href="https://docs.racket-lang.org/scribble/core.html#%28idx._%28gentag._60._%28lib._scribblings%2Fscribble%2Fscribble..scrbl%29%29%29"><code>'exact-chars</code></a> <a href="https://docs.racket-lang.org/scribble/core.html#%28tech._style._property%29">style property</a> renders directly to LaTeX.</p> + +<pre><code>@(define (exact . stuff) + @; the style name "relax" puts a `\relax` no-op in front of the stuff + (make-element (make-style "relax" '(exact-chars)) stuff)) + +@exact|{$\lambda x.\, x$}| +@; ==&gt; \relax{$\lambda x.\, x$} + +@(define ($ . math-stuff) + (apply exact (list "$" math-stuff "$"))) + +@${\lambda x.\, x} +@; ==&gt; \relax{$\lambda x.\, x$}</code></pre> + +<h2 id="creating-a-httpdocsracket-langorgguidemodulesyntaxhtml28parthash-lang29lang-for-a-paper">Creating a <a href="http://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29">#lang</a> for a paper</h2> + +<p>For a Scribble document that is split across multiple files, it can be helpful to make a <code>#lang</code> that <a href="http://blog.racket-lang.org/2017/03/languages-as-dotfiles.html">provides a common environment</a>. Instead of starting each file with a <code>require</code>, e.g.:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require["references.rkt" "helper-functions.rkt" scriblib/figure] + +....</code></pre> + +<p>files can start with a name that describes their common purpose:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang conference-2018-submission + +....</code></pre> + +<p>As a bonus, if the language is defined as a package then the Scribble document can use Racket&rsquo;s dependency management tools:</p> + +<pre><code># to install the paper and interactively install dependencies: +$ cd conference-2018-submission; +$ raco pkg install + +# To check that the paper builds with no dependency issues: +$ raco setup --check-pkg-deps conference-2018-submission + +# To run all unit tests +$ raco test -c conference-2018-submission</code></pre> + +<p>To create a package and language:</p> + +<ol> + <li>Move the Scribble document to a directory with the language name, i.e., <code>conference-2018-submission/</code></li> + <li>Write a simple <code>info.rkt</code> to configure the package</li> + <li>Create a normal Racket module that exports the common environment</li> + <li>Create a <code>conference-2018-submission/lang/reader.rkt</code> module</li></ol> + +<p>Details below. For a full example, visit:</p> + +<ul> + <li><a href="https://gitlab.com/bennn/scribble-acmart-example">https://gitlab.com/bennn/scribble-acmart-example</a></li></ul> + +<hr /> + +<h4 id="conference-2018-submissioninforkt"><code>conference-2018-submission/info.rkt</code></h4> + +<p>This file defines the basic metadata for a package. For more about <code>info.rkt</code>, see: <a href="http://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html">Tutorial: Creating a Package</a>.</p> + +<pre><code>#lang info +(define collection "conference-2018-submission") +(define deps '("base" "scribble-lib" "at-exp-lib")) +(define build-deps '("racket-doc" "scribble-doc")) +(define pkg-desc "Paper for Conference 2018") +(define version "0.1")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionmainrkt"><code>conference-2018-submission/main.rkt</code></h4> + +<p>This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> language, the file &ldquo;references.rkt&rdquo;, and the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library.</p> + +<pre><code>#lang racket/base + +(provide + (all-from-out + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")) + +(require + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionlangreaderrkt"><code>conference-2018-submission/lang/reader.rkt</code></h4> + +<p>This file: (1) tells Racket to use the Scribble reader on <code>#lang conference-2018-submission</code> modules, and (2) wraps the result of such modules in a shape that Scribble expects.</p> + +<pre><code>#lang s-exp scribble/base/reader +conference-2018-submission +#:wrapper1 (lambda (t) (cons 'doc (t)))</code></pre> + +<h2 id="links-to-example-documents">Links to Example Documents</h2> + +<p>These documents use the <code>#lang</code> approach to writing a paper with Scribble. Check their <code>main.rkt</code> for example formatting functions and unit tests, and check the <code>.scrbl</code> files to see how the ideas above look in a larger document.</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018">https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018</a></li> + <li><a href="https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018">https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018</a></li></ul> + +<p>Finally, this repository provides a tool to start a new Scribble document:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gtp-paper">https://pkgd.racket-lang.org/pkgn/package/gtp-paper</a></li></ul> + +<h2 id="further-reading">Further Reading</h2> + +<ul> + <li><a href="https://project.inria.fr/coqexchange/checking-machine-checked-proofs/">Checking Machine-Checked Proofs</a></li></ul> + + On-Stack Replacement + http://prl.ccs.neu.edu/blog/2019/01/28/on-stack-replacement/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-01-28-on-stack-replacement + Mon, 28 Jan 2019 10:29:57 UT + Ming-Ho Yee + +<p>Last semester, I took <a href="https://course.ccs.neu.edu/cs7600/">a course</a> where the final project was to write a survey paper on &ldquo;a topic in the intersection between computer systems and your area.&rdquo; So I wrote about on-stack replacement.</p> +<!-- more--> + +<p><strong>Abstract</strong></p> + +<blockquote> + <p>On-stack replacement (OSR) is a programming language implementation technique that allows a running program to switch to a different version of code. For example, a program could start executing optimized code, and then transfer to and start executing unoptimized code. This was the original use case for OSR, to facilitate debugging of optimized code.</p> + <p>After its original use was established, OSR shifted to a different use case: optimizing programs. OSR allows the run-time system to detect if a program is executing an inefficient loop, recompile and optimize the method that contains the loop, and then transfer control to the newly compiled method. Another strategy is to optimize code based on some assumptions, then, if the assumptions are invalidated at run-time, transfer control back to the original, unoptimized code.</p> + <p>In this survey paper, we study how OSR was first introduced as a means for debugging, how it came to be used for program optimizations, its implementation as a reusable library, and other directions of research.</p></blockquote> + +<p>If you&rsquo;re interested, you can find a copy <a href="/img/cs7600-mhyee-survey-paper-osr.pdf">here</a> or on <a href="https://www.overleaf.com/read/smcmsnksxfdk">Overleaf</a>.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in <a href="http://prl.ccs.neu.edu/blog/2017/03/15/tracing-jits-for-dynamic-languages/">tracing JITs for dynamic languages</a>.</em></p> + + The Behavior of Gradual Types: A User Study + http://prl.ccs.neu.edu/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + Tue, 11 Dec 2018 19:50:33 UT + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + Java and Migratory Typing + http://prl.ccs.neu.edu/blog/2018/12/02/java-and-migratory-typing/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + Sun, 02 Dec 2018 14:41:53 UT + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> + + Turnstile Mailing List + http://prl.ccs.neu.edu/blog/2018/11/30/turnstile-mailing-list/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-11-30-turnstile-mailing-list + Fri, 30 Nov 2018 14:55:30 UT + Stephen Chang + +<p><a href="https://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> now has a mailing list: <a href="https://groups.google.com/forum/#!forum/turnstile-users">https://groups.google.com/forum/#!forum/turnstile-users</a></p> +<!-- more--> + +<p>There is also a <code>#turnstile</code> channel on <a href="https://racket.slack.com">the Racket Slack</a>.</p> + + Disappearing Code + http://prl.ccs.neu.edu/blog/2018/11/24/disappearing-code/?utm_source=all&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-11-24-disappearing-code + Sat, 24 Nov 2018 09:52:58 UT + Ben Greenman + +<p>Two experiences at <a href="https://2018.splashcon.org/home">SPLASH 2018</a> reminded me that software gets thrown away and replaced.</p> +<!-- more--> + +<h3 id="story-1">Story 1</h3> + +<p>The first reminder came near the end of a <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a> by <a href="https://people.csail.mit.edu/rinard/">Martin Rinard</a>. Once upon a time, Martin was working as a consultant and a firm asked him to review a software package. (The firm wanted a second opinion about how the software computed its results.) The firm sent a zipfile; Martin found six versions of the code inside; the firm said &ldquo;well, please check all six versions&rdquo;; and it turned out:</p> + +<ul> + <li><strong>Version 1</strong> : the source code was written in a domain-specific language (DSL) that generated code for the application</li> + <li><strong>Version 2</strong> : the DSL source was the same as version 1, but the generated code was slightly modified</li> + <li>&hellip;</li> + <li><strong>Version 6</strong> : the generated code was the source code and the DSL was gone</li></ul> + +<p>The moral of Martin&rsquo;s story was: (1) the creators of a software system are often different from the maintainers, and (2) researchers need to build tools to help these maintainers.</p> + +<h3 id="story-2">Story 2</h3> + +<p>The second reminder came from a teaching assistant who said the <a href="https://www.cs.cornell.edu/courses/cs3110/2018fa/">functional programming course</a> at their institution was currently using a Python script to test students&rsquo; code. Once upon a time, I was a teaching assistant for the <a href="https://www.cs.cornell.edu/courses/cs3110/2014sp/">same course</a> at the same institution. We had trouble testing students&rsquo; code via the Python script left by the pre&ndash;2013 course staff, so I wrote a <a href="https://gitlab.com/bengreenman/ocaml_tools/">command-line tool</a> to handle the tests and other compile/run/grade tasks. To keep history from repeating itself, I used the same language the course teaches (OCaml) and wrote some documentation &mdash; but it seems like that was not enough. At any rate, writing the tool was a good exercise.</p> + +<blockquote> + <p><em>In the end, everybody must understand for himself.</em> &mdash; <a href="https://dl.acm.org/citation.cfm?id=3731">Per Martin-Löf</a></p></blockquote> + +<h3 id="reflection">Reflection</h3> + +<p>In each story, the maintainers of a software system threw away some old code to make their job easier in the short term. How can we stop this &ldquo;re-inventing the wheel&rdquo; from happening?</p> + +<p>Martin Rinard&rsquo;s solution is to let maintenance programmers keep their current habits, but provide tools to make the short-term, pragmatic solutions into a more robust systems. Search for "<a href="https://people.csail.mit.edu/rinard/paper/osdi04.pdf">failure-oblivious computing</a>" to learn more (this was the topic of his <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a>).</p> + +<p>In Story 1, the maintainers were able to avoid the DSL by modifying an inherited blob of DSL-generated code. If the DSL did not generate code, history might have taken a different course; it might be best to start with a language that offers tools for linguistic re-use, and to build a DSL from these tools &mdash; so there is no generated code. The Racket programming language is exploring this path. For a recent example, see the <a href="https://www2.ccs.neu.edu/racket/pubs/icfp17-acf.pdf">video-lang paper</a>.</p> + +<p>The Story 2 test harness, however, was not generating code. Its maintainers discarded a &ldquo;big&rdquo; program written in a typed functional language in favor of a script. Perhaps we need a language that allows mixing statically-typed and dynamically-typed code (shouts out to <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">my own research</a>).</p> + +<p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p> + +<hr /> + +<h4 id="addendum-comment-from-mitch-wand">Addendum: comment from Mitch Wand</h4> + +<blockquote> + <p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p></blockquote> + +<p>Ermm, this works better for sourdough bread than for people.</p> + +<p>Even in the not-so-real world of checking student solutions, there&rsquo;s often no way of guaranteeing that one half of a pair will be around for the second round. They may be on co-op. Or the course will not be offered the next semster/year/etc. Or the course will change at the next offering (from OCaml to Python or from Racket to Java) so that large chunks of the infrastructure will have to be discarded or rewritten.</p> + +<p>The &ldquo;real&rdquo; solution is to write literate code (as we preached incessantly in PDP), so that the next reader will have at least some clue as about what you wrote. This just may be sufficient incentive to modify rather than rebuild from scratch.</p> + +<p>Ever the optimist, &mdash;Mitch</p> \ No newline at end of file diff --git a/blog/feeds/announcement.atom.xml b/blog/feeds/announcement.atom.xml new file mode 100644 index 00000000..dd1bcd2b --- /dev/null +++ b/blog/feeds/announcement.atom.xml @@ -0,0 +1,20 @@ + + + PRL Blog: Posts tagged 'announcement' + + + urn:http-prl-ccs-neu-edu:-blog-tags-announcement-html + 2018-11-30T14:55:30Z + + Turnstile Mailing List + + urn:http-prl-ccs-neu-edu:-blog-2018-11-30-turnstile-mailing-list + 2018-11-30T14:55:30Z + 2018-11-30T14:55:30Z + + Stephen Chang + +<p><a href="https://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> now has a mailing list: <a href="https://groups.google.com/forum/#!forum/turnstile-users">https://groups.google.com/forum/#!forum/turnstile-users</a></p> +<!-- more--> + +<p>There is also a <code>#turnstile</code> channel on <a href="https://racket.slack.com">the Racket Slack</a>.</p> \ No newline at end of file diff --git a/blog/feeds/announcement.rss.xml b/blog/feeds/announcement.rss.xml new file mode 100644 index 00000000..5fba3ef5 --- /dev/null +++ b/blog/feeds/announcement.rss.xml @@ -0,0 +1,20 @@ + + + + PRL Blog: Posts tagged 'announcement' + PRL Blog: Posts tagged 'announcement' + http://prl.ccs.neu.edu/blog/tags/announcement.html + Fri, 30 Nov 2018 14:55:30 UT + Fri, 30 Nov 2018 14:55:30 UT + 1800 + + Turnstile Mailing List + http://prl.ccs.neu.edu/blog/2018/11/30/turnstile-mailing-list/?utm_source=announcement&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-11-30-turnstile-mailing-list + Fri, 30 Nov 2018 14:55:30 UT + Stephen Chang + +<p><a href="https://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> now has a mailing list: <a href="https://groups.google.com/forum/#!forum/turnstile-users">https://groups.google.com/forum/#!forum/turnstile-users</a></p> +<!-- more--> + +<p>There is also a <code>#turnstile</code> channel on <a href="https://racket.slack.com">the Racket Slack</a>.</p> \ No newline at end of file diff --git a/blog/feeds/array-language.atom.xml b/blog/feeds/array-language.atom.xml new file mode 100644 index 00000000..eef32ca1 --- /dev/null +++ b/blog/feeds/array-language.atom.xml @@ -0,0 +1,54 @@ + + + PRL Blog: Posts tagged 'array language' + + + urn:http-prl-ccs-neu-edu:-blog-tags-array-language-html + 2017-05-04T18:26:48Z + + Rank Polymorphism + + urn:http-prl-ccs-neu-edu:-blog-2017-05-04-rank-polymorphism + 2017-05-04T18:26:48Z + 2017-05-04T18:26:48Z + + Justin Slepak + +<p>Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let&rsquo;s just call it <code>lerp</code>) for scalars:</p> + +<pre><code>(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))</code></pre> + +<p>The number marks on each argument indicate the expected &ldquo;rank&rdquo; of the argument: how many dimensions it should have. In this case, each one is marked <code>0</code>, indicating a scalar (<em>i.e.</em>, 0-dimensional) argument. The function is usable as-is for</p> + +<ul> + <li> + <p>α-blending two RGB pixels</p></li> + <li> + <p>dimming or brightening an image</p></li> + <li> + <p>fade transition between video scenes</p></li></ul> +<!-- more--> + +<p>Each of these use cases mixes the argument dimensions a little differently. A pixel is a vector (a rank&ndash;1 structure) of numbers representing color channel values, so the α-blending case uses two vector arguments and one scalar argument.</p> + +<p>The only real difference between these use cases is the iteration space: they&rsquo;re all effectively loop nests around the same basic scalar operation. In a rank-polymorphic language, the iteration space is derived automatically from the data, so you don&rsquo;t need to write out the control structure yourself.</p> + +<p>The fundamental idea behind function application here is breaking the argument arrays into lower-ranked pieces called &ldquo;cells.&rdquo; Each cell has the rank expected by the function being applied. In the case of <code>lerp</code>, the pixels, images, videos, etc. are all broken up into rank&ndash;0 (scalar) cells because <code>lerp</code> expects rank&ndash;0 arguments. Other expected ranks are possible as well— a vector dot product function <code>dot-prod</code> would call for rank&ndash;1 cells, and a matrix inversion function <code>minv</code> would call for rank&ndash;2 cells.</p> + +<p>The structure built up around the cells is called the &ldquo;frame.&rdquo; A matrix array is a rank&ndash;2 frame containing rank&ndash;0 cells for <code>lerp</code>, but it would be a rank&ndash;1 frame containing rank&ndash;1 cells for <code>dot-prod</code> and a rank&ndash;0 frame containing a single rank&ndash;1 cell for <code>minv</code>. A rank-<em>n</em> array could be broken down into a frame of cells in <em>n+1</em> different ways, and it&rsquo;s the function being applied that determines which decomposition to use.</p> + +<p>Unfortunately, the implicit control structure that&rsquo;s so convenient for the programmer is a problem for a compiler. Historically, implementations of such languages have had to do without static information about the iteration space. Interpreters (and line-at-a-time compilers, to a lesser extent) get to inspect the concrete data they&rsquo;re dealing with, but static compilers have had to make do with emitting a generic loop structure. A &ldquo;loop&rdquo; over a scalar might sound like trivial overhead, but not when it appears within some other hot loop. Being unable to see when loop boundaries match up is also a barrier to loop fusion. The lack of thorough static shape information was a long-standing problem my advisor pointed out to me when I was a new student looking at possible research projects, and he was interested in applying some form of static analysis to gather that information.</p> + +<p>The first step in addressing it was to come up with a formal semantics for rank polymorphism. Although <a href="http://www.jsoftware.com/papers/APL.htm">APL has existed since the 1960s</a>, it had mostly lived in a separate world from mainstream programming language research. The formal techniques developed in PL had seen little to no application to APL and its &ldquo;successor&rdquo; language J.</p> + +<p>There&rsquo;s a lot to dislike about APL and J—special case behavior in many of the primitive operators, limited function arity, syntactic separation of first-order and second-order functions, the impossibility of parsing an entire program at once (fun fact: static analysis <a href="http://dl.acm.org/citation.cfm?id=805380">has been tried</a> there)—and of course the idiosyncratic identifiers used for primops have prompted plenty of internet arguments. None of those things are essential to the programming model, so I&rsquo;m <a href="http://www.ccs.neu.edu/home/jrslepak/proposal.pdf">building a new language called Remora</a> to isolate the aspects I want to study.</p> + +<p>People don&rsquo;t always think of a type system as a form of static analysis, but it turned out to be an effective way of gathering shape information. Remora&rsquo;s <a href="http://www.ccs.neu.edu/home/jrslepak/esop14-full.pdf">type system</a> uses a restricted form of dependent types, in the style of <a href="https://www.cs.cmu.edu/~rwh/theses/xi.pdf">Dependent ML</a>. An array type is indexed by the shape, the numeric sizes of the array&rsquo;s individual dimensions. Index polymorphism (<em>i.e.</em>, Π types) allows functions to work on varying cell shapes and even varying cell ranks (which is essential for primitives like <code>append</code> and <code>reduce</code>, which operate along the major axis of arrays, no matter their rank). Frame-rank polymorphism, which gives rise to the control structure, remains completely implicit, leaving it to be identified by the type rule for function application. As a nice bonus, type soundness rules out run-time errors arising from incompatible argument shapes.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2016/10/19/history-of-actors/">History of Actors</a></li> + <li><a href="http://prl.ccs.neu.edu/blog/2017/02/21/datalog-for-static-analysis/">Datalog for Static Analysis</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/array-language.rss.xml b/blog/feeds/array-language.rss.xml new file mode 100644 index 00000000..48cb4004 --- /dev/null +++ b/blog/feeds/array-language.rss.xml @@ -0,0 +1,54 @@ + + + + PRL Blog: Posts tagged 'array language' + PRL Blog: Posts tagged 'array language' + http://prl.ccs.neu.edu/blog/tags/array-language.html + Thu, 04 May 2017 18:26:48 UT + Thu, 04 May 2017 18:26:48 UT + 1800 + + Rank Polymorphism + http://prl.ccs.neu.edu/blog/2017/05/04/rank-polymorphism/?utm_source=array-language&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-04-rank-polymorphism + Thu, 04 May 2017 18:26:48 UT + Justin Slepak + +<p>Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let&rsquo;s just call it <code>lerp</code>) for scalars:</p> + +<pre><code>(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))</code></pre> + +<p>The number marks on each argument indicate the expected &ldquo;rank&rdquo; of the argument: how many dimensions it should have. In this case, each one is marked <code>0</code>, indicating a scalar (<em>i.e.</em>, 0-dimensional) argument. The function is usable as-is for</p> + +<ul> + <li> + <p>α-blending two RGB pixels</p></li> + <li> + <p>dimming or brightening an image</p></li> + <li> + <p>fade transition between video scenes</p></li></ul> +<!-- more--> + +<p>Each of these use cases mixes the argument dimensions a little differently. A pixel is a vector (a rank&ndash;1 structure) of numbers representing color channel values, so the α-blending case uses two vector arguments and one scalar argument.</p> + +<p>The only real difference between these use cases is the iteration space: they&rsquo;re all effectively loop nests around the same basic scalar operation. In a rank-polymorphic language, the iteration space is derived automatically from the data, so you don&rsquo;t need to write out the control structure yourself.</p> + +<p>The fundamental idea behind function application here is breaking the argument arrays into lower-ranked pieces called &ldquo;cells.&rdquo; Each cell has the rank expected by the function being applied. In the case of <code>lerp</code>, the pixels, images, videos, etc. are all broken up into rank&ndash;0 (scalar) cells because <code>lerp</code> expects rank&ndash;0 arguments. Other expected ranks are possible as well— a vector dot product function <code>dot-prod</code> would call for rank&ndash;1 cells, and a matrix inversion function <code>minv</code> would call for rank&ndash;2 cells.</p> + +<p>The structure built up around the cells is called the &ldquo;frame.&rdquo; A matrix array is a rank&ndash;2 frame containing rank&ndash;0 cells for <code>lerp</code>, but it would be a rank&ndash;1 frame containing rank&ndash;1 cells for <code>dot-prod</code> and a rank&ndash;0 frame containing a single rank&ndash;1 cell for <code>minv</code>. A rank-<em>n</em> array could be broken down into a frame of cells in <em>n+1</em> different ways, and it&rsquo;s the function being applied that determines which decomposition to use.</p> + +<p>Unfortunately, the implicit control structure that&rsquo;s so convenient for the programmer is a problem for a compiler. Historically, implementations of such languages have had to do without static information about the iteration space. Interpreters (and line-at-a-time compilers, to a lesser extent) get to inspect the concrete data they&rsquo;re dealing with, but static compilers have had to make do with emitting a generic loop structure. A &ldquo;loop&rdquo; over a scalar might sound like trivial overhead, but not when it appears within some other hot loop. Being unable to see when loop boundaries match up is also a barrier to loop fusion. The lack of thorough static shape information was a long-standing problem my advisor pointed out to me when I was a new student looking at possible research projects, and he was interested in applying some form of static analysis to gather that information.</p> + +<p>The first step in addressing it was to come up with a formal semantics for rank polymorphism. Although <a href="http://www.jsoftware.com/papers/APL.htm">APL has existed since the 1960s</a>, it had mostly lived in a separate world from mainstream programming language research. The formal techniques developed in PL had seen little to no application to APL and its &ldquo;successor&rdquo; language J.</p> + +<p>There&rsquo;s a lot to dislike about APL and J—special case behavior in many of the primitive operators, limited function arity, syntactic separation of first-order and second-order functions, the impossibility of parsing an entire program at once (fun fact: static analysis <a href="http://dl.acm.org/citation.cfm?id=805380">has been tried</a> there)—and of course the idiosyncratic identifiers used for primops have prompted plenty of internet arguments. None of those things are essential to the programming model, so I&rsquo;m <a href="http://www.ccs.neu.edu/home/jrslepak/proposal.pdf">building a new language called Remora</a> to isolate the aspects I want to study.</p> + +<p>People don&rsquo;t always think of a type system as a form of static analysis, but it turned out to be an effective way of gathering shape information. Remora&rsquo;s <a href="http://www.ccs.neu.edu/home/jrslepak/esop14-full.pdf">type system</a> uses a restricted form of dependent types, in the style of <a href="https://www.cs.cmu.edu/~rwh/theses/xi.pdf">Dependent ML</a>. An array type is indexed by the shape, the numeric sizes of the array&rsquo;s individual dimensions. Index polymorphism (<em>i.e.</em>, Π types) allows functions to work on varying cell shapes and even varying cell ranks (which is essential for primitives like <code>append</code> and <code>reduce</code>, which operate along the major axis of arrays, no matter their rank). Frame-rank polymorphism, which gives rise to the control structure, remains completely implicit, leaving it to be identified by the type rule for function application. As a nice bonus, type soundness rules out run-time errors arising from incompatible argument shapes.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2016/10/19/history-of-actors/">History of Actors</a></li> + <li><a href="http://prl.ccs.neu.edu/blog/2017/02/21/datalog-for-static-analysis/">Datalog for Static Analysis</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/benchmarking.atom.xml b/blog/feeds/benchmarking.atom.xml new file mode 100644 index 00000000..f88daaa9 --- /dev/null +++ b/blog/feeds/benchmarking.atom.xml @@ -0,0 +1,62 @@ + + + PRL Blog: Posts tagged 'benchmarking' + + + urn:http-prl-ccs-neu-edu:-blog-tags-benchmarking-html + 2016-08-03T14:09:02Z + + A few cores too many + + urn:http-prl-ccs-neu-edu:-blog-2016-08-03-a-few-cores-too-many + 2016-08-03T14:09:02Z + 2016-08-03T14:09:02Z + + Ben Greenman + +<p>Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.</p> +<!-- more--> + +<p>Last year, we proposed a method for evaluating the performance of gradual type systems based on measuring <em>every possible configuration</em> of typed and untyped code that a programmer might explore <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">(pdf)</a>. Given the freedom that gradual typing offers, this is the only realistic way to measure the performance of a gradual type system.</p> + +<p>But it is a lot to measure! While developing the method, we spent over 3 months benchmarking a total of 75,844 configurations. Each configuration is a complete program and some gradual typings caused some programs to slow by 50x or even 100x, so many of these configurations took minutes to run.</p> + +<p>The next question we asked was naturally &ldquo;how can we scale this method to large software projects?&rdquo; In <a href="http://docs.racket-lang.org/ts-reference/Libraries_Provided_With_Typed_Racket.html#%28part._.Porting_.Untyped_.Modules_to_.Typed_.Racket%29">our case</a>, the number of gradually typed configurations scaled exponentially with the number of modules. Current gradual type system for <a href="https://github.com/mvitousek/reticulated">Python</a> and <a href="http://www.di.ens.fr/~zappa/readings/ecoop15.pdf">JavaScript</a> are exponential in the number of <em>variables</em> in the program.</p> + +<p>We explored two solutions:</p> + +<ol> + <li>Max New began work on a prediction model (inspired by work on <a href="http://subs.emis.de/LNI/Proceedings/Proceedings213/185.pdf">software product lines</a>) to estimate the performance of <code>2^N</code> configurations after polynomially-many measurements.</li> + <li>Asumu Takikawa and I shopped for a multi-core computer.</li></ol> + +<p>By Thanksgiving, we had bought a Linux machine with 2 <a href="http://www.amd.com/en-us/products/server/opteron/6000/6300">AMD Opteron 6376 2.3GHz</a> processors (16 cores each) and put it to work running benchmarks on 29 cores simultaneously. Life was good.</p> + +<p>Later that winter, Max implemented a prediction algorithm. The basic idea was to focus on <em>boundaries</em> between modules and isolate their effect on performance. If two modules are untyped, their boundary will have zero cost. If the same two modules are typed, their boundary might result in an overall performance improvement due to type-driven optimizations. And if one module is typed and the other untyped, their boundary will suffer some cost of type checking at runtime. In general a program with <code>N</code> modules has at most <code>N(N - 1) / 2</code> internal boundaries, so it is far more time-efficient to measure only the boundaries than to benchmark <code>2^N</code> gradually typed configurations.</p> + +<p>Fast-forward to March, we had a prototype prediction algorithm and it was time to test. Again using 29 cores (because, why not), we gathered cost/benefit numbers for one 4-module benchmark and used them to predict performance for its 16 configurations. The results were not very good.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-1.png" alt="Figure 1: True running time vs. predicted running time for 16 configurations" /> + <p class="caption">Figure 1: True running time vs. predicted running time for 16 configurations</p></div> + +<p>Those green circles are the ground truth, the average running time after 5 iterations of each config. The blue triangles are what we predicted. Except for configurations 0 and 8, the triangles are FAR off from the truth. Many are even negative &hellip; obviously the algorithm needs work.</p> + +<p>But then, out of frustration, desperation, or just good luck, Max compared the predictions to ground truth data gathered on a <em>single</em> core, leaving the other 31 cores idle.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-2.png" alt="Figure 2: Predictions made using measurements from a single core" /> + <p class="caption">Figure 2: Predictions made using measurements from a single core</p></div> + +<p>First off, the red &ldquo;sequential truth&rdquo; dots are slightly closer to the predicted triangles. Second &mdash; and this is the scary part &mdash; the red dots are very different from the green dots. <em>Running on 1 core vs. 29 cores should not change the measurements!</em></p> + +<p>From here we tried increasing the running time of the benchmark, removing I/O and system calls, checking for hyperthreading (ARM cores don&rsquo;t support it), and even changing the cores&rsquo; CPU governor. The hope was that results taken from 1 core could match results from <code>N</code> cores, for some <code>N &gt; 1</code>. It turns out <code>N = 2</code> was stable, but even for <code>N = 3</code> we found graphs like the following:</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-3.png" alt="Figure 3: exact running times. Same-colored dots in each column should be tightly clustered." /> + <p class="caption">Figure 3: exact running times. Same-colored dots in each column should be tightly clustered.</p></div> + +<p>This data is for the same 16 configurations as the previous two graphs. Green dots are exact running times measured with 25 cores. Red dots are running times measured with 1 core. The red dots are much closer together, and always unimodal. The green dots are evidence that maybe the 32-core machine has, as Jan Vitek put it, 30 cores too many.</p> + +<blockquote> + <p>&ldquo;Oh my. You think it&rsquo;ll never happen to you. Well, now I&rsquo;ve learned my lesson.&rdquo;</p><!-- bg: If anyone knows this quote I will be AMAZED. If anyone can even Google this quote, I'll buy them 2 beers and a pizza.--></blockquote> + +<p>And so, we said goodbye to the last 4 months of data and started over running at most two cores. The new results are all stable, but still we keep pinching ourselves.</p> + +<p>P.S. the results from <a href="http://www.ccs.neu.edu/racket/pubs/#popl16-tfgnvf">POPL 2016</a> are just fine, as they were not taken on the new machine running more than 2 cores. If you have time to confirm, that data is in our <a href="http://www.ccs.neu.edu/home/asumu/artifacts/popl-2016/">artifact</a> and in the <a href="https://github.com/nuprl/gradual-typing-performance/tree/master/paper/popl-2016/data">gradual-typing-performance</a> repo.</p> \ No newline at end of file diff --git a/blog/feeds/benchmarking.rss.xml b/blog/feeds/benchmarking.rss.xml new file mode 100644 index 00000000..f57d0724 --- /dev/null +++ b/blog/feeds/benchmarking.rss.xml @@ -0,0 +1,62 @@ + + + + PRL Blog: Posts tagged 'benchmarking' + PRL Blog: Posts tagged 'benchmarking' + http://prl.ccs.neu.edu/blog/tags/benchmarking.html + Wed, 03 Aug 2016 14:09:02 UT + Wed, 03 Aug 2016 14:09:02 UT + 1800 + + A few cores too many + http://prl.ccs.neu.edu/blog/2016/08/03/a-few-cores-too-many/?utm_source=benchmarking&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-08-03-a-few-cores-too-many + Wed, 03 Aug 2016 14:09:02 UT + Ben Greenman + +<p>Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.</p> +<!-- more--> + +<p>Last year, we proposed a method for evaluating the performance of gradual type systems based on measuring <em>every possible configuration</em> of typed and untyped code that a programmer might explore <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">(pdf)</a>. Given the freedom that gradual typing offers, this is the only realistic way to measure the performance of a gradual type system.</p> + +<p>But it is a lot to measure! While developing the method, we spent over 3 months benchmarking a total of 75,844 configurations. Each configuration is a complete program and some gradual typings caused some programs to slow by 50x or even 100x, so many of these configurations took minutes to run.</p> + +<p>The next question we asked was naturally &ldquo;how can we scale this method to large software projects?&rdquo; In <a href="http://docs.racket-lang.org/ts-reference/Libraries_Provided_With_Typed_Racket.html#%28part._.Porting_.Untyped_.Modules_to_.Typed_.Racket%29">our case</a>, the number of gradually typed configurations scaled exponentially with the number of modules. Current gradual type system for <a href="https://github.com/mvitousek/reticulated">Python</a> and <a href="http://www.di.ens.fr/~zappa/readings/ecoop15.pdf">JavaScript</a> are exponential in the number of <em>variables</em> in the program.</p> + +<p>We explored two solutions:</p> + +<ol> + <li>Max New began work on a prediction model (inspired by work on <a href="http://subs.emis.de/LNI/Proceedings/Proceedings213/185.pdf">software product lines</a>) to estimate the performance of <code>2^N</code> configurations after polynomially-many measurements.</li> + <li>Asumu Takikawa and I shopped for a multi-core computer.</li></ol> + +<p>By Thanksgiving, we had bought a Linux machine with 2 <a href="http://www.amd.com/en-us/products/server/opteron/6000/6300">AMD Opteron 6376 2.3GHz</a> processors (16 cores each) and put it to work running benchmarks on 29 cores simultaneously. Life was good.</p> + +<p>Later that winter, Max implemented a prediction algorithm. The basic idea was to focus on <em>boundaries</em> between modules and isolate their effect on performance. If two modules are untyped, their boundary will have zero cost. If the same two modules are typed, their boundary might result in an overall performance improvement due to type-driven optimizations. And if one module is typed and the other untyped, their boundary will suffer some cost of type checking at runtime. In general a program with <code>N</code> modules has at most <code>N(N - 1) / 2</code> internal boundaries, so it is far more time-efficient to measure only the boundaries than to benchmark <code>2^N</code> gradually typed configurations.</p> + +<p>Fast-forward to March, we had a prototype prediction algorithm and it was time to test. Again using 29 cores (because, why not), we gathered cost/benefit numbers for one 4-module benchmark and used them to predict performance for its 16 configurations. The results were not very good.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-1.png" alt="Figure 1: True running time vs. predicted running time for 16 configurations" /> + <p class="caption">Figure 1: True running time vs. predicted running time for 16 configurations</p></div> + +<p>Those green circles are the ground truth, the average running time after 5 iterations of each config. The blue triangles are what we predicted. Except for configurations 0 and 8, the triangles are FAR off from the truth. Many are even negative &hellip; obviously the algorithm needs work.</p> + +<p>But then, out of frustration, desperation, or just good luck, Max compared the predictions to ground truth data gathered on a <em>single</em> core, leaving the other 31 cores idle.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-2.png" alt="Figure 2: Predictions made using measurements from a single core" /> + <p class="caption">Figure 2: Predictions made using measurements from a single core</p></div> + +<p>First off, the red &ldquo;sequential truth&rdquo; dots are slightly closer to the predicted triangles. Second &mdash; and this is the scary part &mdash; the red dots are very different from the green dots. <em>Running on 1 core vs. 29 cores should not change the measurements!</em></p> + +<p>From here we tried increasing the running time of the benchmark, removing I/O and system calls, checking for hyperthreading (ARM cores don&rsquo;t support it), and even changing the cores&rsquo; CPU governor. The hope was that results taken from 1 core could match results from <code>N</code> cores, for some <code>N &gt; 1</code>. It turns out <code>N = 2</code> was stable, but even for <code>N = 3</code> we found graphs like the following:</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-3.png" alt="Figure 3: exact running times. Same-colored dots in each column should be tightly clustered." /> + <p class="caption">Figure 3: exact running times. Same-colored dots in each column should be tightly clustered.</p></div> + +<p>This data is for the same 16 configurations as the previous two graphs. Green dots are exact running times measured with 25 cores. Red dots are running times measured with 1 core. The red dots are much closer together, and always unimodal. The green dots are evidence that maybe the 32-core machine has, as Jan Vitek put it, 30 cores too many.</p> + +<blockquote> + <p>&ldquo;Oh my. You think it&rsquo;ll never happen to you. Well, now I&rsquo;ve learned my lesson.&rdquo;</p><!-- bg: If anyone knows this quote I will be AMAZED. If anyone can even Google this quote, I'll buy them 2 beers and a pizza.--></blockquote> + +<p>And so, we said goodbye to the last 4 months of data and started over running at most two cores. The new results are all stable, but still we keep pinching ourselves.</p> + +<p>P.S. the results from <a href="http://www.ccs.neu.edu/racket/pubs/#popl16-tfgnvf">POPL 2016</a> are just fine, as they were not taken on the new machine running more than 2 cores. If you have time to confirm, that data is in our <a href="http://www.ccs.neu.edu/home/asumu/artifacts/popl-2016/">artifact</a> and in the <a href="https://github.com/nuprl/gradual-typing-performance/tree/master/paper/popl-2016/data">gradual-typing-performance</a> repo.</p> \ No newline at end of file diff --git a/blog/feeds/beta.atom.xml b/blog/feeds/beta.atom.xml new file mode 100644 index 00000000..2798ecb5 --- /dev/null +++ b/blog/feeds/beta.atom.xml @@ -0,0 +1,306 @@ + + + PRL Blog: Posts tagged 'beta' + + + urn:http-prl-ccs-neu-edu:-blog-tags-beta-html + 2016-11-02T21:10:18Z + + Beta Reduction (Part 1) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + 2016-11-02T21:10:18Z + 2016-11-02T21:10:18Z + + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/beta.rss.xml b/blog/feeds/beta.rss.xml new file mode 100644 index 00000000..bd4455e0 --- /dev/null +++ b/blog/feeds/beta.rss.xml @@ -0,0 +1,306 @@ + + + + PRL Blog: Posts tagged 'beta' + PRL Blog: Posts tagged 'beta' + http://prl.ccs.neu.edu/blog/tags/beta.html + Wed, 02 Nov 2016 21:10:18 UT + Wed, 02 Nov 2016 21:10:18 UT + 1800 + + Beta Reduction (Part 1) + http://prl.ccs.neu.edu/blog/2016/11/02/beta-reduction-part-1/?utm_source=beta&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + Wed, 02 Nov 2016 21:10:18 UT + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/books.atom.xml b/blog/feeds/books.atom.xml new file mode 100644 index 00000000..d9f6461d --- /dev/null +++ b/blog/feeds/books.atom.xml @@ -0,0 +1,16 @@ + + + PRL Blog: Posts tagged 'books' + + + urn:http-prl-ccs-neu-edu:-blog-tags-books-html + 2016-11-30T15:24:45Z + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-30-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages + 2016-11-30T15:24:45Z + 2016-11-30T15:24:45Z + + Jonathan Schuster + \ No newline at end of file diff --git a/blog/feeds/books.rss.xml b/blog/feeds/books.rss.xml new file mode 100644 index 00000000..7ce85c27 --- /dev/null +++ b/blog/feeds/books.rss.xml @@ -0,0 +1,16 @@ + + + + PRL Blog: Posts tagged 'books' + PRL Blog: Posts tagged 'books' + http://prl.ccs.neu.edu/blog/tags/books.html + Wed, 30 Nov 2016 15:24:45 UT + Wed, 30 Nov 2016 15:24:45 UT + 1800 + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + http://prl.ccs.neu.edu/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/?utm_source=books&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-30-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages + Wed, 30 Nov 2016 15:24:45 UT + Jonathan Schuster + \ No newline at end of file diff --git a/blog/feeds/bugs.atom.xml b/blog/feeds/bugs.atom.xml new file mode 100644 index 00000000..1e7444e4 --- /dev/null +++ b/blog/feeds/bugs.atom.xml @@ -0,0 +1,37 @@ + + + PRL Blog: Posts tagged 'bugs' + + + urn:http-prl-ccs-neu-edu:-blog-tags-bugs-html + 2017-05-26T17:00:28Z + + Racket 6.9 and Windows 10 Creators Update + + urn:http-prl-ccs-neu-edu:-blog-2017-05-26-racket-6-9-and-windows-10-creators-update + 2017-05-26T17:00:28Z + 2017-05-26T17:00:28Z + + Leif Andersen + +<p><a href="http://racket-lang.org/">Racket</a> 6.9 was released in April and it has been smooth sailing for many people. However, some people using the <a href="https://blogs.windows.com/windowsexperience/2017/04/11/whats-new-in-the-windows-10-creators-update/">Windows 10 Creators Update</a> have been experiencing <a href="https://github.com/racket/racket/issues/1671">crashes</a>, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.</p> + +<p>The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.</p> + +<p>A VirtualBox image with Racket preinstalled can be downloaded here:</p> + +<ul> + <li><a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova</a></li></ul> + +<p>The username and password for this machine are both <code>racket</code>.</p> +<!-- more--> + +<ol> + <li> + <p>The first thing you need to install is virtualization software. In principle it doesn&rsquo;t matter what you install, but for this tutorial, we will use <a href="https://www.virtualbox.org/">VirtualBox</a>. Go to their <a href="https://www.virtualbox.org/wiki/Downloads">downloads</a> page and download and install the version for your platform (most likely Windows).</p></li> + <li> + <p>Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which <a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">you can download here</a>. The rest of this tutorial will assume you are using this image.</p></li> + <li> + <p>Start up VirtualBox and import the virtual machine. You can do this by clicking on <code>File -&gt; Import Appliance</code>. In the dialog, select the image you downloaded and hit <code>Continue</code>. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click <code>Import</code>.</p></li> + <li> + <p>After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit <code>Start</code>. Once started up, you will see DrRacket and Firefox on the VM&rsquo;s desktop.</p></li></ol> \ No newline at end of file diff --git a/blog/feeds/bugs.rss.xml b/blog/feeds/bugs.rss.xml new file mode 100644 index 00000000..589b29bf --- /dev/null +++ b/blog/feeds/bugs.rss.xml @@ -0,0 +1,37 @@ + + + + PRL Blog: Posts tagged 'bugs' + PRL Blog: Posts tagged 'bugs' + http://prl.ccs.neu.edu/blog/tags/bugs.html + Fri, 26 May 2017 17:00:28 UT + Fri, 26 May 2017 17:00:28 UT + 1800 + + Racket 6.9 and Windows 10 Creators Update + http://prl.ccs.neu.edu/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/?utm_source=bugs&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-26-racket-6-9-and-windows-10-creators-update + Fri, 26 May 2017 17:00:28 UT + Leif Andersen + +<p><a href="http://racket-lang.org/">Racket</a> 6.9 was released in April and it has been smooth sailing for many people. However, some people using the <a href="https://blogs.windows.com/windowsexperience/2017/04/11/whats-new-in-the-windows-10-creators-update/">Windows 10 Creators Update</a> have been experiencing <a href="https://github.com/racket/racket/issues/1671">crashes</a>, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.</p> + +<p>The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.</p> + +<p>A VirtualBox image with Racket preinstalled can be downloaded here:</p> + +<ul> + <li><a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova</a></li></ul> + +<p>The username and password for this machine are both <code>racket</code>.</p> +<!-- more--> + +<ol> + <li> + <p>The first thing you need to install is virtualization software. In principle it doesn&rsquo;t matter what you install, but for this tutorial, we will use <a href="https://www.virtualbox.org/">VirtualBox</a>. Go to their <a href="https://www.virtualbox.org/wiki/Downloads">downloads</a> page and download and install the version for your platform (most likely Windows).</p></li> + <li> + <p>Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which <a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">you can download here</a>. The rest of this tutorial will assume you are using this image.</p></li> + <li> + <p>Start up VirtualBox and import the virtual machine. You can do this by clicking on <code>File -&gt; Import Appliance</code>. In the dialog, select the image you downloaded and hit <code>Continue</code>. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click <code>Import</code>.</p></li> + <li> + <p>After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit <code>Start</code>. Once started up, you will see DrRacket and Firefox on the VM&rsquo;s desktop.</p></li></ol> \ No newline at end of file diff --git a/blog/feeds/calculus.atom.xml b/blog/feeds/calculus.atom.xml new file mode 100644 index 00000000..347d9df0 --- /dev/null +++ b/blog/feeds/calculus.atom.xml @@ -0,0 +1,306 @@ + + + PRL Blog: Posts tagged 'calculus' + + + urn:http-prl-ccs-neu-edu:-blog-tags-calculus-html + 2016-11-02T21:10:18Z + + Beta Reduction (Part 1) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + 2016-11-02T21:10:18Z + 2016-11-02T21:10:18Z + + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/calculus.rss.xml b/blog/feeds/calculus.rss.xml new file mode 100644 index 00000000..34e0ea94 --- /dev/null +++ b/blog/feeds/calculus.rss.xml @@ -0,0 +1,306 @@ + + + + PRL Blog: Posts tagged 'calculus' + PRL Blog: Posts tagged 'calculus' + http://prl.ccs.neu.edu/blog/tags/calculus.html + Wed, 02 Nov 2016 21:10:18 UT + Wed, 02 Nov 2016 21:10:18 UT + 1800 + + Beta Reduction (Part 1) + http://prl.ccs.neu.edu/blog/2016/11/02/beta-reduction-part-1/?utm_source=calculus&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + Wed, 02 Nov 2016 21:10:18 UT + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/castle.atom.xml b/blog/feeds/castle.atom.xml new file mode 100644 index 00000000..f36ce208 --- /dev/null +++ b/blog/feeds/castle.atom.xml @@ -0,0 +1,45 @@ + + + PRL Blog: Posts tagged 'castle' + + + urn:http-prl-ccs-neu-edu:-blog-tags-castle-html + 2019-03-09T14:40:16Z + + PLISS: Learn About PL Implementation in a Castle + + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + 2019-03-09T14:40:16Z + 2019-03-09T14:40:16Z + + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> \ No newline at end of file diff --git a/blog/feeds/castle.rss.xml b/blog/feeds/castle.rss.xml new file mode 100644 index 00000000..25ef8827 --- /dev/null +++ b/blog/feeds/castle.rss.xml @@ -0,0 +1,45 @@ + + + + PRL Blog: Posts tagged 'castle' + PRL Blog: Posts tagged 'castle' + http://prl.ccs.neu.edu/blog/tags/castle.html + Sat, 09 Mar 2019 14:40:16 UT + Sat, 09 Mar 2019 14:40:16 UT + 1800 + + PLISS: Learn About PL Implementation in a Castle + http://prl.ccs.neu.edu/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/?utm_source=castle&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + Sat, 09 Mar 2019 14:40:16 UT + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> \ No newline at end of file diff --git a/blog/feeds/category-theory.atom.xml b/blog/feeds/category-theory.atom.xml new file mode 100644 index 00000000..31b5afb9 --- /dev/null +++ b/blog/feeds/category-theory.atom.xml @@ -0,0 +1,535 @@ + + + PRL Blog: Posts tagged 'category theory' + + + urn:http-prl-ccs-neu-edu:-blog-tags-category-theory-html + 2017-09-27T15:44:57Z + + Final Algebra Semantics is Observational Equivalence + + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + 2017-09-27T15:44:57Z + 2017-09-27T15:44:57Z + + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> + + Closure Conversion as CoYoneda + + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + 2017-08-28T10:30:00Z + 2017-08-28T10:30:00Z + + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> + + Categorical Semantics for Dynamically Typed Programming Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + 2017-05-01T12:25:17Z + 2017-05-01T12:25:17Z + + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> + + Understanding Constructive Galois Connections + + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + 2016-11-16T00:00:00Z + 2016-11-16T00:00:00Z + + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/category-theory.rss.xml b/blog/feeds/category-theory.rss.xml new file mode 100644 index 00000000..f81820cf --- /dev/null +++ b/blog/feeds/category-theory.rss.xml @@ -0,0 +1,529 @@ + + + + PRL Blog: Posts tagged 'category theory' + PRL Blog: Posts tagged 'category theory' + http://prl.ccs.neu.edu/blog/tags/category-theory.html + Wed, 27 Sep 2017 15:44:57 UT + Wed, 27 Sep 2017 15:44:57 UT + 1800 + + Final Algebra Semantics is Observational Equivalence + http://prl.ccs.neu.edu/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/?utm_source=category-theory&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + Wed, 27 Sep 2017 15:44:57 UT + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> + + Closure Conversion as CoYoneda + http://prl.ccs.neu.edu/blog/2017/08/28/closure-conversion-as-coyoneda/?utm_source=category-theory&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + Mon, 28 Aug 2017 10:30:00 UT + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> + + Categorical Semantics for Dynamically Typed Programming Languages + http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/?utm_source=category-theory&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + Mon, 01 May 2017 12:25:17 UT + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> + + Understanding Constructive Galois Connections + http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/?utm_source=category-theory&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + Wed, 16 Nov 2016 00:00:00 UT + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/clojure.atom.xml b/blog/feeds/clojure.atom.xml new file mode 100644 index 00000000..dfd8f46a --- /dev/null +++ b/blog/feeds/clojure.atom.xml @@ -0,0 +1,252 @@ + + + PRL Blog: Posts tagged 'clojure' + + + urn:http-prl-ccs-neu-edu:-blog-tags-clojure-html + 2022-01-06T17:56:08Z + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + 2022-01-06T17:56:08Z + 2022-01-06T17:56:08Z + + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> \ No newline at end of file diff --git a/blog/feeds/clojure.rss.xml b/blog/feeds/clojure.rss.xml new file mode 100644 index 00000000..30055ee9 --- /dev/null +++ b/blog/feeds/clojure.rss.xml @@ -0,0 +1,252 @@ + + + + PRL Blog: Posts tagged 'clojure' + PRL Blog: Posts tagged 'clojure' + http://prl.ccs.neu.edu/blog/tags/clojure.html + Thu, 06 Jan 2022 17:56:08 UT + Thu, 06 Jan 2022 17:56:08 UT + 1800 + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + http://prl.ccs.neu.edu/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/?utm_source=clojure&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + Thu, 06 Jan 2022 17:56:08 UT + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> \ No newline at end of file diff --git a/blog/feeds/clojurescript.atom.xml b/blog/feeds/clojurescript.atom.xml new file mode 100644 index 00000000..71263b24 --- /dev/null +++ b/blog/feeds/clojurescript.atom.xml @@ -0,0 +1,252 @@ + + + PRL Blog: Posts tagged 'clojurescript' + + + urn:http-prl-ccs-neu-edu:-blog-tags-clojurescript-html + 2022-01-06T17:56:08Z + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + 2022-01-06T17:56:08Z + 2022-01-06T17:56:08Z + + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> \ No newline at end of file diff --git a/blog/feeds/clojurescript.rss.xml b/blog/feeds/clojurescript.rss.xml new file mode 100644 index 00000000..8a0c6208 --- /dev/null +++ b/blog/feeds/clojurescript.rss.xml @@ -0,0 +1,252 @@ + + + + PRL Blog: Posts tagged 'clojurescript' + PRL Blog: Posts tagged 'clojurescript' + http://prl.ccs.neu.edu/blog/tags/clojurescript.html + Thu, 06 Jan 2022 17:56:08 UT + Thu, 06 Jan 2022 17:56:08 UT + 1800 + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + http://prl.ccs.neu.edu/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/?utm_source=clojurescript&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + Thu, 06 Jan 2022 17:56:08 UT + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> \ No newline at end of file diff --git a/blog/feeds/closure-conversion.atom.xml b/blog/feeds/closure-conversion.atom.xml new file mode 100644 index 00000000..58245d06 --- /dev/null +++ b/blog/feeds/closure-conversion.atom.xml @@ -0,0 +1,164 @@ + + + PRL Blog: Posts tagged 'closure conversion' + + + urn:http-prl-ccs-neu-edu:-blog-tags-closure-conversion-html + 2017-08-28T10:30:00Z + + Closure Conversion as CoYoneda + + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + 2017-08-28T10:30:00Z + 2017-08-28T10:30:00Z + + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/closure-conversion.rss.xml b/blog/feeds/closure-conversion.rss.xml new file mode 100644 index 00000000..4d810933 --- /dev/null +++ b/blog/feeds/closure-conversion.rss.xml @@ -0,0 +1,164 @@ + + + + PRL Blog: Posts tagged 'closure conversion' + PRL Blog: Posts tagged 'closure conversion' + http://prl.ccs.neu.edu/blog/tags/closure-conversion.html + Mon, 28 Aug 2017 10:30:00 UT + Mon, 28 Aug 2017 10:30:00 UT + 1800 + + Closure Conversion as CoYoneda + http://prl.ccs.neu.edu/blog/2017/08/28/closure-conversion-as-coyoneda/?utm_source=closure-conversion&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + Mon, 28 Aug 2017 10:30:00 UT + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/coYoneda.atom.xml b/blog/feeds/coYoneda.atom.xml new file mode 100644 index 00000000..57cc4321 --- /dev/null +++ b/blog/feeds/coYoneda.atom.xml @@ -0,0 +1,164 @@ + + + PRL Blog: Posts tagged 'coYoneda' + + + urn:http-prl-ccs-neu-edu:-blog-tags-coYoneda-html + 2017-08-28T10:30:00Z + + Closure Conversion as CoYoneda + + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + 2017-08-28T10:30:00Z + 2017-08-28T10:30:00Z + + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/coYoneda.rss.xml b/blog/feeds/coYoneda.rss.xml new file mode 100644 index 00000000..a7061b85 --- /dev/null +++ b/blog/feeds/coYoneda.rss.xml @@ -0,0 +1,164 @@ + + + + PRL Blog: Posts tagged 'coYoneda' + PRL Blog: Posts tagged 'coYoneda' + http://prl.ccs.neu.edu/blog/tags/coYoneda.html + Mon, 28 Aug 2017 10:30:00 UT + Mon, 28 Aug 2017 10:30:00 UT + 1800 + + Closure Conversion as CoYoneda + http://prl.ccs.neu.edu/blog/2017/08/28/closure-conversion-as-coyoneda/?utm_source=coYoneda&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + Mon, 28 Aug 2017 10:30:00 UT + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/compiler-correctness.atom.xml b/blog/feeds/compiler-correctness.atom.xml new file mode 100644 index 00000000..88acfb5f --- /dev/null +++ b/blog/feeds/compiler-correctness.atom.xml @@ -0,0 +1,45 @@ + + + PRL Blog: Posts tagged 'compiler correctness' + + + urn:http-prl-ccs-neu-edu:-blog-tags-compiler-correctness-html + 2016-10-11T17:41:16Z + + CompCert Overview + + urn:http-prl-ccs-neu-edu:-blog-2016-10-11-compcert-overview + 2016-10-11T17:41:16Z + 2016-10-11T17:41:16Z + + Ben Greenman + +<p>If you are interested in learning about the <em>internals</em> of the CompCert C compiler but would rather not read its source code, this post is for you.</p> +<!-- more--> + +<p>(This is a public service announcement.)</p> + +<p>Last fall, I gave a short lecture on the 2006 paper <a href="http://gallium.inria.fr/~xleroy/publi/compiler-certif.pdf">&ldquo;Formal Certification of a Compiler Back-End&rdquo;</a> by Xavier Leroy for Amal Ahmed&rsquo;s <a href="http://www.ccs.neu.edu/home/amal/course/7480-f15/">&ldquo;Special Topics in Programming Languages&rdquo;</a> class. Rather than present CompCert as it existed in 2006, I read the documentation and source code for <a href="https://github.com/AbsInt/CompCert/releases/tag/v2.5">CompCert 2.5</a> (released June 2015). The lecture then focused on three questions:</p> + +<ul> + <li>What subset of C does CompCert handle, today?</li> + <li>What optimizing passes does CompCert perform?</li> + <li>What is the &ldquo;correctness theorem&rdquo; for CompCert, and what does this theorem mean?</li></ul> + +<p>My notes for the lecture give a &ldquo;mid-level&rdquo; summary of the compiler &mdash; there are more details than you&rsquo;ll find in papers, but it&rsquo;s (hopefully!) easier to read than the source code. The document is also hyperlinked to locations in the <a href="https://github.com/AbsInt/CompCert">CompCert GitHub repository</a>.</p> + +<p>Here is the document:</p> + +<blockquote> + <p> <a href="http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf">http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf</a></p></blockquote> + +<p>And here is a table-of-contents:</p> + +<ol> + <li>Motivation, details of the source and target languages, high-level guarantees</li> + <li>Compiler pipeline, optimizing passes, links intermediate language grammars and Coq theorems</li> + <li>Background on compiler correctness</li> + <li>CompCert&rsquo;s correctness, properties that CompCert does <strong>not</strong> guarantee</li> + <li>Recent (2006 &ndash; 2015) work in the CompCert ecosystem</li></ol> + +<p>The document ends with a short description of two other research projects that have grown into &ldquo;industry software&rdquo; and a link to Xaver Leroy&rsquo;s <a href="https://www.cs.uoregon.edu/research/summerschool/summer12/curriculum.html">OPLSS lectures on certified compilers</a>. Enjoy!</p> \ No newline at end of file diff --git a/blog/feeds/compiler-correctness.rss.xml b/blog/feeds/compiler-correctness.rss.xml new file mode 100644 index 00000000..072f1811 --- /dev/null +++ b/blog/feeds/compiler-correctness.rss.xml @@ -0,0 +1,45 @@ + + + + PRL Blog: Posts tagged 'compiler correctness' + PRL Blog: Posts tagged 'compiler correctness' + http://prl.ccs.neu.edu/blog/tags/compiler-correctness.html + Tue, 11 Oct 2016 17:41:16 UT + Tue, 11 Oct 2016 17:41:16 UT + 1800 + + CompCert Overview + http://prl.ccs.neu.edu/blog/2016/10/11/compcert-overview/?utm_source=compiler-correctness&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-11-compcert-overview + Tue, 11 Oct 2016 17:41:16 UT + Ben Greenman + +<p>If you are interested in learning about the <em>internals</em> of the CompCert C compiler but would rather not read its source code, this post is for you.</p> +<!-- more--> + +<p>(This is a public service announcement.)</p> + +<p>Last fall, I gave a short lecture on the 2006 paper <a href="http://gallium.inria.fr/~xleroy/publi/compiler-certif.pdf">&ldquo;Formal Certification of a Compiler Back-End&rdquo;</a> by Xavier Leroy for Amal Ahmed&rsquo;s <a href="http://www.ccs.neu.edu/home/amal/course/7480-f15/">&ldquo;Special Topics in Programming Languages&rdquo;</a> class. Rather than present CompCert as it existed in 2006, I read the documentation and source code for <a href="https://github.com/AbsInt/CompCert/releases/tag/v2.5">CompCert 2.5</a> (released June 2015). The lecture then focused on three questions:</p> + +<ul> + <li>What subset of C does CompCert handle, today?</li> + <li>What optimizing passes does CompCert perform?</li> + <li>What is the &ldquo;correctness theorem&rdquo; for CompCert, and what does this theorem mean?</li></ul> + +<p>My notes for the lecture give a &ldquo;mid-level&rdquo; summary of the compiler &mdash; there are more details than you&rsquo;ll find in papers, but it&rsquo;s (hopefully!) easier to read than the source code. The document is also hyperlinked to locations in the <a href="https://github.com/AbsInt/CompCert">CompCert GitHub repository</a>.</p> + +<p>Here is the document:</p> + +<blockquote> + <p> <a href="http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf">http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf</a></p></blockquote> + +<p>And here is a table-of-contents:</p> + +<ol> + <li>Motivation, details of the source and target languages, high-level guarantees</li> + <li>Compiler pipeline, optimizing passes, links intermediate language grammars and Coq theorems</li> + <li>Background on compiler correctness</li> + <li>CompCert&rsquo;s correctness, properties that CompCert does <strong>not</strong> guarantee</li> + <li>Recent (2006 &ndash; 2015) work in the CompCert ecosystem</li></ol> + +<p>The document ends with a short description of two other research projects that have grown into &ldquo;industry software&rdquo; and a link to Xaver Leroy&rsquo;s <a href="https://www.cs.uoregon.edu/research/summerschool/summer12/curriculum.html">OPLSS lectures on certified compilers</a>. Enjoy!</p> \ No newline at end of file diff --git a/blog/feeds/compilers.atom.xml b/blog/feeds/compilers.atom.xml new file mode 100644 index 00000000..1ec1ec38 --- /dev/null +++ b/blog/feeds/compilers.atom.xml @@ -0,0 +1,164 @@ + + + PRL Blog: Posts tagged 'compilers' + + + urn:http-prl-ccs-neu-edu:-blog-tags-compilers-html + 2017-08-28T10:30:00Z + + Closure Conversion as CoYoneda + + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + 2017-08-28T10:30:00Z + 2017-08-28T10:30:00Z + + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/compilers.rss.xml b/blog/feeds/compilers.rss.xml new file mode 100644 index 00000000..9a0230b0 --- /dev/null +++ b/blog/feeds/compilers.rss.xml @@ -0,0 +1,164 @@ + + + + PRL Blog: Posts tagged 'compilers' + PRL Blog: Posts tagged 'compilers' + http://prl.ccs.neu.edu/blog/tags/compilers.html + Mon, 28 Aug 2017 10:30:00 UT + Mon, 28 Aug 2017 10:30:00 UT + 1800 + + Closure Conversion as CoYoneda + http://prl.ccs.neu.edu/blog/2017/08/28/closure-conversion-as-coyoneda/?utm_source=compilers&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + Mon, 28 Aug 2017 10:30:00 UT + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> \ No newline at end of file diff --git a/blog/feeds/complete-monitoring.atom.xml b/blog/feeds/complete-monitoring.atom.xml new file mode 100644 index 00000000..f20a72f3 --- /dev/null +++ b/blog/feeds/complete-monitoring.atom.xml @@ -0,0 +1,105 @@ + + + PRL Blog: Posts tagged 'complete monitoring' + + + urn:http-prl-ccs-neu-edu:-blog-tags-complete-monitoring-html + 2019-10-31T21:58:26Z + + Complete Monitors for Gradual Types + + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + 2019-10-31T21:58:26Z + 2019-10-31T21:58:26Z + + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> \ No newline at end of file diff --git a/blog/feeds/complete-monitoring.rss.xml b/blog/feeds/complete-monitoring.rss.xml new file mode 100644 index 00000000..c8437e96 --- /dev/null +++ b/blog/feeds/complete-monitoring.rss.xml @@ -0,0 +1,105 @@ + + + + PRL Blog: Posts tagged 'complete monitoring' + PRL Blog: Posts tagged 'complete monitoring' + http://prl.ccs.neu.edu/blog/tags/complete-monitoring.html + Thu, 31 Oct 2019 21:58:26 UT + Thu, 31 Oct 2019 21:58:26 UT + 1800 + + Complete Monitors for Gradual Types + http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/?utm_source=complete-monitoring&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + Thu, 31 Oct 2019 21:58:26 UT + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> \ No newline at end of file diff --git a/blog/feeds/conference.atom.xml b/blog/feeds/conference.atom.xml new file mode 100644 index 00000000..a5082867 --- /dev/null +++ b/blog/feeds/conference.atom.xml @@ -0,0 +1,69 @@ + + + PRL Blog: Posts tagged 'conference' + + + urn:http-prl-ccs-neu-edu:-blog-tags-conference-html + 2016-09-15T21:18:45Z + + NEPLS on October 7th at Northeastern University + + urn:http-prl-ccs-neu-edu:-blog-2016-09-15-nepls-on-october-7th-at-northeastern-university + 2016-09-15T21:18:45Z + 2016-09-15T21:18:45Z + + Ben Greenman + +<p>The next edition of the New England Programming Language Seminar (NEPLS) will be held on Friday, October 7th at Northeastern University. Organizers are Gabriel Scherer and Max New. See you there!</p> +<!-- more--> + +<p>Here is the official announcement from the NEPLS mailing list.</p> + +<blockquote> + <p>Hi everyone,</p> + <p>The next New England Programming Languages and Systems Symposium will take place on</p> + <blockquote> + <p> Friday, October 7th 2016</p></blockquote> + <p>at</p> + <blockquote> + <p> Northeastern University, Boston.</p></blockquote> + <p>Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is</p> + <blockquote> + <p> Sunday, September 25th.</p></blockquote> + <p>Acceptance notifications will be out on Thursday, September 28th.</p> + <p>Send your proposal to</p> + <blockquote> + <p> <a href="mailto:nepls-talks-committee@lists.cs.brown.edu">nepls-talks-committee@lists.cs.brown.edu</a></p></blockquote> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <blockquote> + <p> <a href="http://www.nepls.org/">http://www.nepls.org/</a></p></blockquote></blockquote> + +<p>To subscribe to the NEPLS mailing list, visit this page:</p> + +<p><a href="https://lists.cs.brown.edu/sympa/subscribe/nepls">https://lists.cs.brown.edu/sympa/subscribe/nepls</a></p> + + NEPLS on May 31st at UMass, Amherst + + urn:http-prl-ccs-neu-edu:-blog-2016-05-03-nepls-on-may-31st-at-umass-amherst + 2016-05-03T08:21:07Z + 2016-05-03T08:21:07Z + + Gabriel Scherer + +<p>It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!</p> +<!-- more--> + +<blockquote> + <p>The next New England Programming Languages and Systems Symposium will take place on Tuesday, May 31st 2016 at University of Massachusetts, Amherst. Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is Tuesday, May 17th. Send your proposal to talks@nepls.org.</p> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <p> http://www.nepls.org/</p></blockquote> + +<p>I&rsquo;m personally fond of such regional events, which is a great time to learn about the research around us in a less formal and exhausting setting than a 200-attendees conference.</p> + +<p>If you are in the area, please consider applying to talk about your work. If one of your colleague is working on something you find exciting, please invite them to apply!</p> \ No newline at end of file diff --git a/blog/feeds/conference.rss.xml b/blog/feeds/conference.rss.xml new file mode 100644 index 00000000..b12348cc --- /dev/null +++ b/blog/feeds/conference.rss.xml @@ -0,0 +1,67 @@ + + + + PRL Blog: Posts tagged 'conference' + PRL Blog: Posts tagged 'conference' + http://prl.ccs.neu.edu/blog/tags/conference.html + Thu, 15 Sep 2016 21:18:45 UT + Thu, 15 Sep 2016 21:18:45 UT + 1800 + + NEPLS on October 7th at Northeastern University + http://prl.ccs.neu.edu/blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/?utm_source=conference&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-09-15-nepls-on-october-7th-at-northeastern-university + Thu, 15 Sep 2016 21:18:45 UT + Ben Greenman + +<p>The next edition of the New England Programming Language Seminar (NEPLS) will be held on Friday, October 7th at Northeastern University. Organizers are Gabriel Scherer and Max New. See you there!</p> +<!-- more--> + +<p>Here is the official announcement from the NEPLS mailing list.</p> + +<blockquote> + <p>Hi everyone,</p> + <p>The next New England Programming Languages and Systems Symposium will take place on</p> + <blockquote> + <p> Friday, October 7th 2016</p></blockquote> + <p>at</p> + <blockquote> + <p> Northeastern University, Boston.</p></blockquote> + <p>Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is</p> + <blockquote> + <p> Sunday, September 25th.</p></blockquote> + <p>Acceptance notifications will be out on Thursday, September 28th.</p> + <p>Send your proposal to</p> + <blockquote> + <p> <a href="mailto:nepls-talks-committee@lists.cs.brown.edu">nepls-talks-committee@lists.cs.brown.edu</a></p></blockquote> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <blockquote> + <p> <a href="http://www.nepls.org/">http://www.nepls.org/</a></p></blockquote></blockquote> + +<p>To subscribe to the NEPLS mailing list, visit this page:</p> + +<p><a href="https://lists.cs.brown.edu/sympa/subscribe/nepls">https://lists.cs.brown.edu/sympa/subscribe/nepls</a></p> + + NEPLS on May 31st at UMass, Amherst + http://prl.ccs.neu.edu/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/?utm_source=conference&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-03-nepls-on-may-31st-at-umass-amherst + Tue, 03 May 2016 08:21:07 UT + Gabriel Scherer + +<p>It is my pleasure to relay the following announcement for the next edition of the New England Programming Language Seminer (NEPLS), to be held on Tuesday May 31st at UMass, Amherst, organized by Arjun Guha. Venez nombreux!</p> +<!-- more--> + +<blockquote> + <p>The next New England Programming Languages and Systems Symposium will take place on Tuesday, May 31st 2016 at University of Massachusetts, Amherst. Please mark it in your calendars!</p> + <p>The speaker selection committee solicits talks for this meeting. To propose yourself or someone else, send a title, list of authors, and a brief description. You may provide UP TO ONE PAGE of description, but you can keep it as short as a paragraph. We particularly invite talks by researchers from outside the area who are visiting on the date of the NEPLS meeting.</p> + <p>Talks can vary in length. Though 30-minute conference-style slots are traditional, speakers may request slots of as little as 5 minutes; we encourage the shorter formats. This variety permits the presentation of smaller results, preliminary work, progress reports on ongoing projects (such as language standards and compiler toolkits), and updates to past presentations. In general, NEPLS talks need not sound like conference presentations.</p> + <p>The submission deadline is Tuesday, May 17th. Send your proposal to talks@nepls.org.</p> + <p>More details about NEPLS are available on the NEPLS webpage:</p> + <p> http://www.nepls.org/</p></blockquote> + +<p>I&rsquo;m personally fond of such regional events, which is a great time to learn about the research around us in a less formal and exhausting setting than a 200-attendees conference.</p> + +<p>If you are in the area, please consider applying to talk about your work. If one of your colleague is working on something you find exciting, please invite them to apply!</p> \ No newline at end of file diff --git a/blog/feeds/constructions.atom.xml b/blog/feeds/constructions.atom.xml new file mode 100644 index 00000000..2aa09677 --- /dev/null +++ b/blog/feeds/constructions.atom.xml @@ -0,0 +1,73 @@ + + + PRL Blog: Posts tagged 'constructions' + + + urn:http-prl-ccs-neu-edu:-blog-tags-constructions-html + 2016-10-31T17:20:33Z + + Meaningful Distinctions + + urn:http-prl-ccs-neu-edu:-blog-2016-10-31-meaningful-distinctions + 2016-10-31T17:20:33Z + 2016-10-31T17:20:33Z + + Ben Greenman + +<blockquote> + <p>&ldquo;Meaningful distinctions deserve to be maintained.&rdquo; &mdash; Errett A. Bishop</p></blockquote> + +<p>Likewise, memorable quotations deserve to be read in context. In this spirit, I am happy to present the above &ldquo;basic principle&rdquo; in its context: <a href="/img/sicm.pdf"><em>Schizophrenia in contemporary mathematics</em> (pdf)</a></p> + +<p>Read on for a brief summary.</p> +<!-- more--> + +<hr /> + +<p>I first read the above quotation in <a href="http://www.michaelbeeson.com/research/papers/BishopForeword.pdf">Michael Beeson&rsquo;s introduction</a> to the 2012 edition of Bishop&rsquo;s <a href="https://www.amazon.com/Foundations-Constructive-Analysis-Errett-Bishop/dp/4871877140"><em>Foundations of Constructive Analysis</em></a>. That was two years ago.</p> + +<p>Last month, I tried to find its context. <a href="https://books.google.com/books?id=uPx8tGCaxzUC&amp;pg=PA214&amp;lpg=PA214&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=cWjwOTnNuT&amp;sig=wN143wNyfXtMFLGABBQM-22aSOQ&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIHjAA#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">Many</a> <a href="https://www.jstor.org/stable/2589553">other</a> <a href="https://books.google.com/books?id=J4DkBwAAQBAJ&amp;pg=PA6&amp;lpg=PA6&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=KYkrkBrJd_&amp;sig=AAK1A_uIkQlVcYCY1TFljfA3CqA&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIJTAC#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">uses</a> <a href="https://books.google.com/books?id=oN5nsPkXhhsC&amp;pg=PR6&amp;lpg=PR6&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=4doTufVdsy&amp;sig=u3e_Z_xdN-tjt9p1eqQ88juA5Ns&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIKDAD#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">of</a> <a href="https://books.google.com/books?id=GR44SKXCZJsC&amp;pg=RA1-PA199&amp;lpg=RA1-PA199&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=lNpzR5QV7h&amp;sig=IGg2Q_KtreSAhrbSJxsV7mQ8xok&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIMDAF#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">the</a> <a href="http://www.math.canterbury.ac.nz/php/groups/cm/faq/">quote</a> <a href="http://www.ben-sherman.net/aux/evident-logic.pdf">cited</a> a <em>Schizophrenia in comtemporary mathematics</em>, but I could not find an electronic copy. (It turns out, the AMS Bookstore page for <a href="http://bookstore.ams.org/conm-39"><em>Erret Bishop: Reflections on Him and His Research</em></a> includes a facsimile.)</p> + +<p>Lest anyone else be tempted to conjure the ancient magic of inter-library loan, here is a scan of the pages I borrowed. Thanks to the University of Toledo for supplying the hard copy.</p> + +<blockquote> + <p> <a href="/img/sicm.pdf">prl.ccs.neu.edu/img/sicm.pdf</a></p></blockquote> + +<p>The document is Bishop&rsquo;s &ldquo;feeling for the philosophical issues involved&rdquo; in constructive mathematics. First, Bishop lists &ldquo;schizophrenic attributes&rdquo; (trouble spots) of contemporary mathematics. Next, he gives basic principles of constructivism and Brouwer&rsquo;s interpretation of the logical quantifiers. Along the way, and as a source of examples, Bishop describes integers, sets, and real numbers. The emphasis is always on common-sense meaning and finite constructions.</p> + +<p>After a brief summary and reflection, the last ten pages list recent advances in constructive mathematics and upcoming tasks. The open tasks are particularly interesting:</p> + +<ul> + <li>systematically develop (constructive) algebra</li> + <li>give a constructive foundation for general topology</li> + <li>engage with the deeper &ldquo;meaning of mathematics&rdquo;</li></ul> + +<p>The popular quote on &ldquo;Meaningful Distinctions&rdquo; appears early in the paper, as one of Bishop&rsquo;s four principles that &ldquo;stand out as basic&rdquo; to the philosophy of constructivism:</p> + +<blockquote> + <p>A. Mathematics is common sense.</p> + <p>B. Do not ask whether a statement is true until you know what it means.</p> + <p>C. A proof is any completely convincing argument.</p> + <p>D. Meaningful distinctions deserve to be maintained.</p></blockquote> + +<p>I had no idea that D was &ldquo;a principle&rdquo;, or that it had three siblings.</p> + +<p>To further tempt you into reading the whole truth, here are some of my favorite phrases:</p> + +<blockquote> + <ul> + <li>One suspects that the majority of pure mathematicians &hellip; ignore as much content as they possibly can.</li> + <li>We have geared ourselves to producing research mathematicians who will begin to write papers as soon as possible. This anti-social and anti-intellectual process defeats even its own narrow ends.</li> + <li>&hellip; truth is not a source of trouble to the constructivist, because of his emphasis on meaning.</li> + <li>&hellip; guided primarily by considerations of content rather than elegance and formal attractiveness &hellip;</li> + <li>Let me tell you what a smart sequence will do.</li> + <li>Classical mathematics fails to observe meaningful distinctions having to do with integers.</li> + <li>Constructive mathematics does not postulate a pre-existent universe, with objects lying around waiting to be collected and grouped into sets, like shells on a beach.</li> + <li>It might be worthwhile to investigate the possibility that constructive mathematics would afford a solid philosophical basis for the theory of computation &hellip;</li> + <li>&hellip; if the product of two real numbers is 0, we are not entitled to conclude that one of them is 0.</li> + <li>It is fair to say that almost nobody finds his proof intelligible.</li> + <li>Mathematics is such a complicated activity that disagreements are bound to arise.</li> + <li>Algebraic topology, at least on the elementary level, should not be too difficult to constructivize.</li> + <li>I hope all this accords with your common sense, as it does with mine.</li></ul></blockquote> + +<p>Now go find their context!</p> \ No newline at end of file diff --git a/blog/feeds/constructions.rss.xml b/blog/feeds/constructions.rss.xml new file mode 100644 index 00000000..089f02be --- /dev/null +++ b/blog/feeds/constructions.rss.xml @@ -0,0 +1,73 @@ + + + + PRL Blog: Posts tagged 'constructions' + PRL Blog: Posts tagged 'constructions' + http://prl.ccs.neu.edu/blog/tags/constructions.html + Mon, 31 Oct 2016 17:20:33 UT + Mon, 31 Oct 2016 17:20:33 UT + 1800 + + Meaningful Distinctions + http://prl.ccs.neu.edu/blog/2016/10/31/meaningful-distinctions/?utm_source=constructions&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-31-meaningful-distinctions + Mon, 31 Oct 2016 17:20:33 UT + Ben Greenman + +<blockquote> + <p>&ldquo;Meaningful distinctions deserve to be maintained.&rdquo; &mdash; Errett A. Bishop</p></blockquote> + +<p>Likewise, memorable quotations deserve to be read in context. In this spirit, I am happy to present the above &ldquo;basic principle&rdquo; in its context: <a href="/img/sicm.pdf"><em>Schizophrenia in contemporary mathematics</em> (pdf)</a></p> + +<p>Read on for a brief summary.</p> +<!-- more--> + +<hr /> + +<p>I first read the above quotation in <a href="http://www.michaelbeeson.com/research/papers/BishopForeword.pdf">Michael Beeson&rsquo;s introduction</a> to the 2012 edition of Bishop&rsquo;s <a href="https://www.amazon.com/Foundations-Constructive-Analysis-Errett-Bishop/dp/4871877140"><em>Foundations of Constructive Analysis</em></a>. That was two years ago.</p> + +<p>Last month, I tried to find its context. <a href="https://books.google.com/books?id=uPx8tGCaxzUC&amp;pg=PA214&amp;lpg=PA214&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=cWjwOTnNuT&amp;sig=wN143wNyfXtMFLGABBQM-22aSOQ&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIHjAA#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">Many</a> <a href="https://www.jstor.org/stable/2589553">other</a> <a href="https://books.google.com/books?id=J4DkBwAAQBAJ&amp;pg=PA6&amp;lpg=PA6&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=KYkrkBrJd_&amp;sig=AAK1A_uIkQlVcYCY1TFljfA3CqA&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIJTAC#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">uses</a> <a href="https://books.google.com/books?id=oN5nsPkXhhsC&amp;pg=PR6&amp;lpg=PR6&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=4doTufVdsy&amp;sig=u3e_Z_xdN-tjt9p1eqQ88juA5Ns&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIKDAD#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">of</a> <a href="https://books.google.com/books?id=GR44SKXCZJsC&amp;pg=RA1-PA199&amp;lpg=RA1-PA199&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=lNpzR5QV7h&amp;sig=IGg2Q_KtreSAhrbSJxsV7mQ8xok&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIMDAF#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">the</a> <a href="http://www.math.canterbury.ac.nz/php/groups/cm/faq/">quote</a> <a href="http://www.ben-sherman.net/aux/evident-logic.pdf">cited</a> a <em>Schizophrenia in comtemporary mathematics</em>, but I could not find an electronic copy. (It turns out, the AMS Bookstore page for <a href="http://bookstore.ams.org/conm-39"><em>Erret Bishop: Reflections on Him and His Research</em></a> includes a facsimile.)</p> + +<p>Lest anyone else be tempted to conjure the ancient magic of inter-library loan, here is a scan of the pages I borrowed. Thanks to the University of Toledo for supplying the hard copy.</p> + +<blockquote> + <p> <a href="/img/sicm.pdf">prl.ccs.neu.edu/img/sicm.pdf</a></p></blockquote> + +<p>The document is Bishop&rsquo;s &ldquo;feeling for the philosophical issues involved&rdquo; in constructive mathematics. First, Bishop lists &ldquo;schizophrenic attributes&rdquo; (trouble spots) of contemporary mathematics. Next, he gives basic principles of constructivism and Brouwer&rsquo;s interpretation of the logical quantifiers. Along the way, and as a source of examples, Bishop describes integers, sets, and real numbers. The emphasis is always on common-sense meaning and finite constructions.</p> + +<p>After a brief summary and reflection, the last ten pages list recent advances in constructive mathematics and upcoming tasks. The open tasks are particularly interesting:</p> + +<ul> + <li>systematically develop (constructive) algebra</li> + <li>give a constructive foundation for general topology</li> + <li>engage with the deeper &ldquo;meaning of mathematics&rdquo;</li></ul> + +<p>The popular quote on &ldquo;Meaningful Distinctions&rdquo; appears early in the paper, as one of Bishop&rsquo;s four principles that &ldquo;stand out as basic&rdquo; to the philosophy of constructivism:</p> + +<blockquote> + <p>A. Mathematics is common sense.</p> + <p>B. Do not ask whether a statement is true until you know what it means.</p> + <p>C. A proof is any completely convincing argument.</p> + <p>D. Meaningful distinctions deserve to be maintained.</p></blockquote> + +<p>I had no idea that D was &ldquo;a principle&rdquo;, or that it had three siblings.</p> + +<p>To further tempt you into reading the whole truth, here are some of my favorite phrases:</p> + +<blockquote> + <ul> + <li>One suspects that the majority of pure mathematicians &hellip; ignore as much content as they possibly can.</li> + <li>We have geared ourselves to producing research mathematicians who will begin to write papers as soon as possible. This anti-social and anti-intellectual process defeats even its own narrow ends.</li> + <li>&hellip; truth is not a source of trouble to the constructivist, because of his emphasis on meaning.</li> + <li>&hellip; guided primarily by considerations of content rather than elegance and formal attractiveness &hellip;</li> + <li>Let me tell you what a smart sequence will do.</li> + <li>Classical mathematics fails to observe meaningful distinctions having to do with integers.</li> + <li>Constructive mathematics does not postulate a pre-existent universe, with objects lying around waiting to be collected and grouped into sets, like shells on a beach.</li> + <li>It might be worthwhile to investigate the possibility that constructive mathematics would afford a solid philosophical basis for the theory of computation &hellip;</li> + <li>&hellip; if the product of two real numbers is 0, we are not entitled to conclude that one of them is 0.</li> + <li>It is fair to say that almost nobody finds his proof intelligible.</li> + <li>Mathematics is such a complicated activity that disagreements are bound to arise.</li> + <li>Algebraic topology, at least on the elementary level, should not be too difficult to constructivize.</li> + <li>I hope all this accords with your common sense, as it does with mine.</li></ul></blockquote> + +<p>Now go find their context!</p> \ No newline at end of file diff --git a/blog/feeds/coq.atom.xml b/blog/feeds/coq.atom.xml new file mode 100644 index 00000000..dfa1a16d --- /dev/null +++ b/blog/feeds/coq.atom.xml @@ -0,0 +1,176 @@ + + + PRL Blog: Posts tagged 'coq' + + + urn:http-prl-ccs-neu-edu:-blog-tags-coq-html + 2017-02-21T19:04:28Z + + Bullets are good for your Coq proofs + + urn:http-prl-ccs-neu-edu:-blog-2017-02-21-bullets-are-good-for-your-coq-proofs + 2017-02-21T19:04:28Z + 2017-02-21T19:04:28Z + + Gabriel Scherer + +<p>I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.</p> + +<p>Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.</p> +<!-- more--> + +<h2 id="why-bullets">Why bullets</h2> + +<p>While you are doing a proof, Coq shows a list of subgoals that have to be proved before the whole proof is complete. Most proof steps will operate on the current active subgoal, changing the hypotheses or the goal to prove, but some proof steps will split it into several subgoals (growing the total list of goals), or may terminate the proof of the current subgoal and show you the next active subgoal.</p> + +<p>Before bullets, a typical proof script would contain the proofs of each subgoal, one after another.</p> + +<pre><code>induction foo. (* this creates many subgoal *) + +proof of first subgoal. + +proof of second subgoal.</code></pre> + +<p>There are many ways to structure this to make the structure more apparent: people would typically have a comment on each subgoal, or make disciplined use of indentation and blank lines. But, in my experience, a major problem with this style was maintainability in the face of changes to the definitions or parts of automation. It could be very hard of what was happening when a proof suddenly broke after a change before in the file:</p> + +<ul> + <li> + <p>If a proof step now proves <em>less</em> things, then what used to be the end of a subgoal may not be anymore. Coq would then start reading the proof of the next subgoal and try to apply it to the unfinished previous goals, generating very confusing errors (you believe you are in the second subgoal, but the context talks about a leaf case of the first goal).</p></li> + <li> + <p>If a proof step now proves <em>more</em> things, it is also very bad! The next proof steps, meant for the first subgoal (for example), would then apply to the beginning of the second subgoal, and you get very confusing errors again.</p></li></ul> + +<p>What we need for robustness is a way to indicate our <em>intent</em> to Coq, when we think that a subgoal is finished and that a new subgoal starts, so that Coq can fail loudly at the moment where it notices that this intent does not match reality, instead of at an arbitrary later time.</p> + +<p>(The <code>S*Case</code> tactics used in (older versions of) Software Foundations can solve this problem if used in a carefully, systematic way, and additionally provides naming. Alexandre Pilkiewicz implemented an even more powerful <a href="https://github.com/pilki/cases">cases</a> plugin. Bullets are available in standard Coq since 8.4 (released in 2012), and can be used with no effort.)</p> + +<p>There is not much discussion of bullets around; see the <a href="https://coq.inria.fr/distrib/8.6/refman/Reference-Manual009.html#sec326">documentation</a> in the Coq manual. I learned a lot from Arthur Azevedo de Amorim&rsquo;s <a href="https://github.com/arthuraa/poleiro/blob/master/theories/Bullets.v">Bullets.v</a> file.</p> + +<p>Finally, some people don&rsquo;t use bullets, because they systematically use so much automation that they never see subgoals &mdash; each lemma has a one-line proof. This is also a valid style. (I have been going to Adam Chlipala&rsquo;s <a href="https://frap.csail.mit.edu/main">Formal Reasoning about Programs</a> 2017 class, where Adam ignores bullets because that is his usual style.) Because I am not crushy enough to do this from the start, my proofs tend to start with cases and subgoals, and then I refine them to add more automation for robustness. I found bullets very useful for the first step, and during the refinement process.</p> + +<h2 id="bullets">Bullets</h2> + +<p>Bullets are actually a combination of two features, braces <code>{ ... }</code> and actual list bullets &mdash; <code>-</code>, <code>+</code>, <code>*</code>, or homogeneous repetitions of those, for example <code>--</code> or <code>***</code>.</p> + +<h3 id="braces">Braces</h3> + +<p>The opening brace <code>{</code> focuses the proof on the current subgoal. If you finish the proof of the subgoal, the following subgoal will not become accessible automatically; you have to use the closing brace <code>}</code> first. (If you finish the goal earlier than you think, you get an error.) Conversely, <code>}</code> fails if the subgoal is not complete. (If you fail to finish, you get an error.)</p> + +<p>The previous example can thus be written as follows, and will be more robust:</p> + +<pre><code>induction foo. (* this creates many subgoal *) +{ + proof of first subgoal. +} +{ + proof of second subgoal. +}</code></pre> + +<p>If you also want to make sure that an error occurs if the number of subgoals changes (for example if new constructors are added to the inductive type of <code>foo</code>), you can use an outer layer of braces:</p> + +<pre><code>{ induction foo. (* this creates many subgoal *) + { + proof of first subgoal. + } + { + proof of second subgoal. + } +} (* would fail if a new subgoal appeared *)</code></pre> + +<h3 id="list-bullets">List bullets</h3> + +<p>A bullet, for example <code>--</code>, also focuses on the next subgoal. The difference is that when the subgoal is finished, you do not have a closing construction, you must use the same bullet to move to the next subgoal. (Again, this fails if the first proof step changes to prove too much or too little.) With bullets you would write</p> + +<pre><code>induction foo. (* this creates many subgoal *) ++ proof of first subgoal. ++ proof of second subgoal.</code></pre> + +<p>Bullets can be nested, but you must use different bullets for the different nesting levels. For example, if this proof is only one subgoal of a larger proof, you can use:</p> + +<pre><code>- induction foo. (* this creates many subgoal *) + + proof of first subgoal. + + proof of second subgoal. +- (* would fail if a new subgoal appeared *) + rest of the proof</code></pre> + +<p>The natural ordering of bullets, I think, is by increasing number of lines: <code>-</code>, <code>+</code> then <code>*</code> (and then multi-character bullets, I guess). You can also mix bullets with braces: the opening brace resets the bullet scope, any bullet can be used again with the subgoal.</p> + +<p>This gives a large space of freedom in how you want to use these features. You can use only braces, only bullets, braces and only one level of bullets, etc. My own style evolved with experience using the feature, and I will present the current status below.</p> + +<h2 id="my-current-bullet-style">My current bullet style</h2> + +<p>When deciding how to use bullets, one distinguishes the commands that preserve the number of subgoals and those that may create new subgoals. I use some additional distinctions.</p> + +<p>Some tactics, for example <code>assert</code>, create a number of subgoals that is <em>statically</em> known, always the same for the tactic. I then use braces around each sub-proof, except the last one, which I think of as the &ldquo;rest&rdquo; of the current proof.</p> + +<pre><code>assert foo as H. +{ proof of foo. } +rest of the proof using H:foo.</code></pre> + +<p>(If the proof of <code>foo</code> takes several lines, I two-indent them, with the braces alone on their lines.)</p> + +<p>Most tactics create a <em>dynamic</em> number of subgoals, that depends on the specifics of the objects being operated on; this is the case of <code>case</code>, <code>destruct</code>, <code>induction</code> for example. In this case, I open a brace before the tactic, and use a bullet for each subgoal.</p> + +<pre><code>{ induction foo; simpl; auto. +- proof of first remaining subgoal. +- proof of second remaining subgoal. + rest of the proof of the second subgoal. +}</code></pre> + +<p>(Notice that the subgoal-creating step is vertically aligned with the proof steps: I use both braces and bullets, but take only one indentation level each time.)</p> + +<p>As an exception, I may omit the braces if we are at the toplevel of the proof (<code>Proof .. Qed</code> serve as braces).</p> + +<p>Note that omitting the braces here and using different bullets when you nest is also just fine. In my experience it gives proofs that are a bit more pleasant to read but also a bit more cumbersome to edit and move around.</p> + +<p>Finally, a not-uncommon mode of use of &ldquo;dynamic&rdquo; tactics in the sense above is to expect all the cases, except one, to be discharged by direct automation (for example they are all absurd except one). When it is my intent that all cases but one be discharged (and not a coincidence), I express it by not using braces (this command preserves the number of subgoals), but marking the remaining subgoal with a new bullet <em>without</em> increasing the indentation level.</p> + +<pre><code>{ induction foo. +- first subgoal. +- second subgoal. + case blah; discharge all sub-subgoals but one. ++ remaining sub-subgoal of the second subgoal. + finish the sub-subgoal. +- third subgoal. +}</code></pre> + +<p>(This is the only time where I use several bullet levels.)</p> + +<p>If you are the kind of programmer that is passionate about indentation style, I should now have tricked you to use bullets to propose a different variant. Otherwise, please consider using bullets anyway, for example by following the style above, it will make your life easier in the face of changing developments.</p> + + CompCert Overview + + urn:http-prl-ccs-neu-edu:-blog-2016-10-11-compcert-overview + 2016-10-11T17:41:16Z + 2016-10-11T17:41:16Z + + Ben Greenman + +<p>If you are interested in learning about the <em>internals</em> of the CompCert C compiler but would rather not read its source code, this post is for you.</p> +<!-- more--> + +<p>(This is a public service announcement.)</p> + +<p>Last fall, I gave a short lecture on the 2006 paper <a href="http://gallium.inria.fr/~xleroy/publi/compiler-certif.pdf">&ldquo;Formal Certification of a Compiler Back-End&rdquo;</a> by Xavier Leroy for Amal Ahmed&rsquo;s <a href="http://www.ccs.neu.edu/home/amal/course/7480-f15/">&ldquo;Special Topics in Programming Languages&rdquo;</a> class. Rather than present CompCert as it existed in 2006, I read the documentation and source code for <a href="https://github.com/AbsInt/CompCert/releases/tag/v2.5">CompCert 2.5</a> (released June 2015). The lecture then focused on three questions:</p> + +<ul> + <li>What subset of C does CompCert handle, today?</li> + <li>What optimizing passes does CompCert perform?</li> + <li>What is the &ldquo;correctness theorem&rdquo; for CompCert, and what does this theorem mean?</li></ul> + +<p>My notes for the lecture give a &ldquo;mid-level&rdquo; summary of the compiler &mdash; there are more details than you&rsquo;ll find in papers, but it&rsquo;s (hopefully!) easier to read than the source code. The document is also hyperlinked to locations in the <a href="https://github.com/AbsInt/CompCert">CompCert GitHub repository</a>.</p> + +<p>Here is the document:</p> + +<blockquote> + <p> <a href="http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf">http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf</a></p></blockquote> + +<p>And here is a table-of-contents:</p> + +<ol> + <li>Motivation, details of the source and target languages, high-level guarantees</li> + <li>Compiler pipeline, optimizing passes, links intermediate language grammars and Coq theorems</li> + <li>Background on compiler correctness</li> + <li>CompCert&rsquo;s correctness, properties that CompCert does <strong>not</strong> guarantee</li> + <li>Recent (2006 &ndash; 2015) work in the CompCert ecosystem</li></ol> + +<p>The document ends with a short description of two other research projects that have grown into &ldquo;industry software&rdquo; and a link to Xaver Leroy&rsquo;s <a href="https://www.cs.uoregon.edu/research/summerschool/summer12/curriculum.html">OPLSS lectures on certified compilers</a>. Enjoy!</p> \ No newline at end of file diff --git a/blog/feeds/coq.rss.xml b/blog/feeds/coq.rss.xml new file mode 100644 index 00000000..94b48347 --- /dev/null +++ b/blog/feeds/coq.rss.xml @@ -0,0 +1,174 @@ + + + + PRL Blog: Posts tagged 'coq' + PRL Blog: Posts tagged 'coq' + http://prl.ccs.neu.edu/blog/tags/coq.html + Tue, 21 Feb 2017 19:04:28 UT + Tue, 21 Feb 2017 19:04:28 UT + 1800 + + Bullets are good for your Coq proofs + http://prl.ccs.neu.edu/blog/2017/02/21/bullets-are-good-for-your-coq-proofs/?utm_source=coq&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-21-bullets-are-good-for-your-coq-proofs + Tue, 21 Feb 2017 19:04:28 UT + Gabriel Scherer + +<p>I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.</p> + +<p>Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.</p> +<!-- more--> + +<h2 id="why-bullets">Why bullets</h2> + +<p>While you are doing a proof, Coq shows a list of subgoals that have to be proved before the whole proof is complete. Most proof steps will operate on the current active subgoal, changing the hypotheses or the goal to prove, but some proof steps will split it into several subgoals (growing the total list of goals), or may terminate the proof of the current subgoal and show you the next active subgoal.</p> + +<p>Before bullets, a typical proof script would contain the proofs of each subgoal, one after another.</p> + +<pre><code>induction foo. (* this creates many subgoal *) + +proof of first subgoal. + +proof of second subgoal.</code></pre> + +<p>There are many ways to structure this to make the structure more apparent: people would typically have a comment on each subgoal, or make disciplined use of indentation and blank lines. But, in my experience, a major problem with this style was maintainability in the face of changes to the definitions or parts of automation. It could be very hard of what was happening when a proof suddenly broke after a change before in the file:</p> + +<ul> + <li> + <p>If a proof step now proves <em>less</em> things, then what used to be the end of a subgoal may not be anymore. Coq would then start reading the proof of the next subgoal and try to apply it to the unfinished previous goals, generating very confusing errors (you believe you are in the second subgoal, but the context talks about a leaf case of the first goal).</p></li> + <li> + <p>If a proof step now proves <em>more</em> things, it is also very bad! The next proof steps, meant for the first subgoal (for example), would then apply to the beginning of the second subgoal, and you get very confusing errors again.</p></li></ul> + +<p>What we need for robustness is a way to indicate our <em>intent</em> to Coq, when we think that a subgoal is finished and that a new subgoal starts, so that Coq can fail loudly at the moment where it notices that this intent does not match reality, instead of at an arbitrary later time.</p> + +<p>(The <code>S*Case</code> tactics used in (older versions of) Software Foundations can solve this problem if used in a carefully, systematic way, and additionally provides naming. Alexandre Pilkiewicz implemented an even more powerful <a href="https://github.com/pilki/cases">cases</a> plugin. Bullets are available in standard Coq since 8.4 (released in 2012), and can be used with no effort.)</p> + +<p>There is not much discussion of bullets around; see the <a href="https://coq.inria.fr/distrib/8.6/refman/Reference-Manual009.html#sec326">documentation</a> in the Coq manual. I learned a lot from Arthur Azevedo de Amorim&rsquo;s <a href="https://github.com/arthuraa/poleiro/blob/master/theories/Bullets.v">Bullets.v</a> file.</p> + +<p>Finally, some people don&rsquo;t use bullets, because they systematically use so much automation that they never see subgoals &mdash; each lemma has a one-line proof. This is also a valid style. (I have been going to Adam Chlipala&rsquo;s <a href="https://frap.csail.mit.edu/main">Formal Reasoning about Programs</a> 2017 class, where Adam ignores bullets because that is his usual style.) Because I am not crushy enough to do this from the start, my proofs tend to start with cases and subgoals, and then I refine them to add more automation for robustness. I found bullets very useful for the first step, and during the refinement process.</p> + +<h2 id="bullets">Bullets</h2> + +<p>Bullets are actually a combination of two features, braces <code>{ ... }</code> and actual list bullets &mdash; <code>-</code>, <code>+</code>, <code>*</code>, or homogeneous repetitions of those, for example <code>--</code> or <code>***</code>.</p> + +<h3 id="braces">Braces</h3> + +<p>The opening brace <code>{</code> focuses the proof on the current subgoal. If you finish the proof of the subgoal, the following subgoal will not become accessible automatically; you have to use the closing brace <code>}</code> first. (If you finish the goal earlier than you think, you get an error.) Conversely, <code>}</code> fails if the subgoal is not complete. (If you fail to finish, you get an error.)</p> + +<p>The previous example can thus be written as follows, and will be more robust:</p> + +<pre><code>induction foo. (* this creates many subgoal *) +{ + proof of first subgoal. +} +{ + proof of second subgoal. +}</code></pre> + +<p>If you also want to make sure that an error occurs if the number of subgoals changes (for example if new constructors are added to the inductive type of <code>foo</code>), you can use an outer layer of braces:</p> + +<pre><code>{ induction foo. (* this creates many subgoal *) + { + proof of first subgoal. + } + { + proof of second subgoal. + } +} (* would fail if a new subgoal appeared *)</code></pre> + +<h3 id="list-bullets">List bullets</h3> + +<p>A bullet, for example <code>--</code>, also focuses on the next subgoal. The difference is that when the subgoal is finished, you do not have a closing construction, you must use the same bullet to move to the next subgoal. (Again, this fails if the first proof step changes to prove too much or too little.) With bullets you would write</p> + +<pre><code>induction foo. (* this creates many subgoal *) ++ proof of first subgoal. ++ proof of second subgoal.</code></pre> + +<p>Bullets can be nested, but you must use different bullets for the different nesting levels. For example, if this proof is only one subgoal of a larger proof, you can use:</p> + +<pre><code>- induction foo. (* this creates many subgoal *) + + proof of first subgoal. + + proof of second subgoal. +- (* would fail if a new subgoal appeared *) + rest of the proof</code></pre> + +<p>The natural ordering of bullets, I think, is by increasing number of lines: <code>-</code>, <code>+</code> then <code>*</code> (and then multi-character bullets, I guess). You can also mix bullets with braces: the opening brace resets the bullet scope, any bullet can be used again with the subgoal.</p> + +<p>This gives a large space of freedom in how you want to use these features. You can use only braces, only bullets, braces and only one level of bullets, etc. My own style evolved with experience using the feature, and I will present the current status below.</p> + +<h2 id="my-current-bullet-style">My current bullet style</h2> + +<p>When deciding how to use bullets, one distinguishes the commands that preserve the number of subgoals and those that may create new subgoals. I use some additional distinctions.</p> + +<p>Some tactics, for example <code>assert</code>, create a number of subgoals that is <em>statically</em> known, always the same for the tactic. I then use braces around each sub-proof, except the last one, which I think of as the &ldquo;rest&rdquo; of the current proof.</p> + +<pre><code>assert foo as H. +{ proof of foo. } +rest of the proof using H:foo.</code></pre> + +<p>(If the proof of <code>foo</code> takes several lines, I two-indent them, with the braces alone on their lines.)</p> + +<p>Most tactics create a <em>dynamic</em> number of subgoals, that depends on the specifics of the objects being operated on; this is the case of <code>case</code>, <code>destruct</code>, <code>induction</code> for example. In this case, I open a brace before the tactic, and use a bullet for each subgoal.</p> + +<pre><code>{ induction foo; simpl; auto. +- proof of first remaining subgoal. +- proof of second remaining subgoal. + rest of the proof of the second subgoal. +}</code></pre> + +<p>(Notice that the subgoal-creating step is vertically aligned with the proof steps: I use both braces and bullets, but take only one indentation level each time.)</p> + +<p>As an exception, I may omit the braces if we are at the toplevel of the proof (<code>Proof .. Qed</code> serve as braces).</p> + +<p>Note that omitting the braces here and using different bullets when you nest is also just fine. In my experience it gives proofs that are a bit more pleasant to read but also a bit more cumbersome to edit and move around.</p> + +<p>Finally, a not-uncommon mode of use of &ldquo;dynamic&rdquo; tactics in the sense above is to expect all the cases, except one, to be discharged by direct automation (for example they are all absurd except one). When it is my intent that all cases but one be discharged (and not a coincidence), I express it by not using braces (this command preserves the number of subgoals), but marking the remaining subgoal with a new bullet <em>without</em> increasing the indentation level.</p> + +<pre><code>{ induction foo. +- first subgoal. +- second subgoal. + case blah; discharge all sub-subgoals but one. ++ remaining sub-subgoal of the second subgoal. + finish the sub-subgoal. +- third subgoal. +}</code></pre> + +<p>(This is the only time where I use several bullet levels.)</p> + +<p>If you are the kind of programmer that is passionate about indentation style, I should now have tricked you to use bullets to propose a different variant. Otherwise, please consider using bullets anyway, for example by following the style above, it will make your life easier in the face of changing developments.</p> + + CompCert Overview + http://prl.ccs.neu.edu/blog/2016/10/11/compcert-overview/?utm_source=coq&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-11-compcert-overview + Tue, 11 Oct 2016 17:41:16 UT + Ben Greenman + +<p>If you are interested in learning about the <em>internals</em> of the CompCert C compiler but would rather not read its source code, this post is for you.</p> +<!-- more--> + +<p>(This is a public service announcement.)</p> + +<p>Last fall, I gave a short lecture on the 2006 paper <a href="http://gallium.inria.fr/~xleroy/publi/compiler-certif.pdf">&ldquo;Formal Certification of a Compiler Back-End&rdquo;</a> by Xavier Leroy for Amal Ahmed&rsquo;s <a href="http://www.ccs.neu.edu/home/amal/course/7480-f15/">&ldquo;Special Topics in Programming Languages&rdquo;</a> class. Rather than present CompCert as it existed in 2006, I read the documentation and source code for <a href="https://github.com/AbsInt/CompCert/releases/tag/v2.5">CompCert 2.5</a> (released June 2015). The lecture then focused on three questions:</p> + +<ul> + <li>What subset of C does CompCert handle, today?</li> + <li>What optimizing passes does CompCert perform?</li> + <li>What is the &ldquo;correctness theorem&rdquo; for CompCert, and what does this theorem mean?</li></ul> + +<p>My notes for the lecture give a &ldquo;mid-level&rdquo; summary of the compiler &mdash; there are more details than you&rsquo;ll find in papers, but it&rsquo;s (hopefully!) easier to read than the source code. The document is also hyperlinked to locations in the <a href="https://github.com/AbsInt/CompCert">CompCert GitHub repository</a>.</p> + +<p>Here is the document:</p> + +<blockquote> + <p> <a href="http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf">http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf</a></p></blockquote> + +<p>And here is a table-of-contents:</p> + +<ol> + <li>Motivation, details of the source and target languages, high-level guarantees</li> + <li>Compiler pipeline, optimizing passes, links intermediate language grammars and Coq theorems</li> + <li>Background on compiler correctness</li> + <li>CompCert&rsquo;s correctness, properties that CompCert does <strong>not</strong> guarantee</li> + <li>Recent (2006 &ndash; 2015) work in the CompCert ecosystem</li></ol> + +<p>The document ends with a short description of two other research projects that have grown into &ldquo;industry software&rdquo; and a link to Xaver Leroy&rsquo;s <a href="https://www.cs.uoregon.edu/research/summerschool/summer12/curriculum.html">OPLSS lectures on certified compilers</a>. Enjoy!</p> \ No newline at end of file diff --git a/blog/feeds/dawn-of-the-digital-era.atom.xml b/blog/feeds/dawn-of-the-digital-era.atom.xml new file mode 100644 index 00000000..e40ee0cd --- /dev/null +++ b/blog/feeds/dawn-of-the-digital-era.atom.xml @@ -0,0 +1,30 @@ + + + PRL Blog: Posts tagged 'dawn of the digital era' + + + urn:http-prl-ccs-neu-edu:-blog-tags-dawn-of-the-digital-era-html + 2016-06-13T10:50:14Z + + Does anyone still care about printed proceedings? (Grab some at NEU this week!) + + urn:http-prl-ccs-neu-edu:-blog-2016-06-13-does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week + 2016-06-13T10:50:14Z + 2016-06-13T10:50:14Z + + Gabriel Scherer + +<p>Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!</p> +<!-- more--> + +<p>If you are in the area and are interested, feel free to send me an email and come grab them. We have ASPLOS from XIII to XX, PLDI from 2005 to 2015 (but not 2014), and OOPSLA from 2002 to 2015. When I saw the stack, I grabbed the POPL ones from 2002 to 2015, but in fact I have no idea what to do with them and I&rsquo;m a bit skeptical they would be of use to me; if you know you would use them, I would be glad to let you have them.</p> + +<p>If you were to buy those proceedings at conference-subscription rates today, it would cost you a small fortune. Yet nobody seems to want them. An odd disconnect, that I found amusing and maybe worthy of a blog post.</p> + +<p>But don&rsquo;t get me wrong, the future of printed proceedings is not an important question. We should rather be asking ourselves: why are the products of the work of our communities not easily accessible in an Open Access long-term archive? Are you submitting your articles to arXiv, or another archive? Why not?</p> + +<p>Not caring about printed proceedings is perfectly fine; but please care about people outside institutions that want to access your work &mdash; for example, master student myself.</p> + +<hr /> + +<p><em>Update (August 2017):</em> Gabriel returned to France and left the POPL proceedings at his desk. The proceedings are sitting in the hallway at NEU, in case anyone cares.</p> \ No newline at end of file diff --git a/blog/feeds/dawn-of-the-digital-era.rss.xml b/blog/feeds/dawn-of-the-digital-era.rss.xml new file mode 100644 index 00000000..bb37c4ea --- /dev/null +++ b/blog/feeds/dawn-of-the-digital-era.rss.xml @@ -0,0 +1,30 @@ + + + + PRL Blog: Posts tagged 'dawn of the digital era' + PRL Blog: Posts tagged 'dawn of the digital era' + http://prl.ccs.neu.edu/blog/tags/dawn-of-the-digital-era.html + Mon, 13 Jun 2016 10:50:14 UT + Mon, 13 Jun 2016 10:50:14 UT + 1800 + + Does anyone still care about printed proceedings? (Grab some at NEU this week!) + http://prl.ccs.neu.edu/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/?utm_source=dawn-of-the-digital-era&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-13-does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week + Mon, 13 Jun 2016 10:50:14 UT + Gabriel Scherer + +<p>Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!</p> +<!-- more--> + +<p>If you are in the area and are interested, feel free to send me an email and come grab them. We have ASPLOS from XIII to XX, PLDI from 2005 to 2015 (but not 2014), and OOPSLA from 2002 to 2015. When I saw the stack, I grabbed the POPL ones from 2002 to 2015, but in fact I have no idea what to do with them and I&rsquo;m a bit skeptical they would be of use to me; if you know you would use them, I would be glad to let you have them.</p> + +<p>If you were to buy those proceedings at conference-subscription rates today, it would cost you a small fortune. Yet nobody seems to want them. An odd disconnect, that I found amusing and maybe worthy of a blog post.</p> + +<p>But don&rsquo;t get me wrong, the future of printed proceedings is not an important question. We should rather be asking ourselves: why are the products of the work of our communities not easily accessible in an Open Access long-term archive? Are you submitting your articles to arXiv, or another archive? Why not?</p> + +<p>Not caring about printed proceedings is perfectly fine; but please care about people outside institutions that want to access your work &mdash; for example, master student myself.</p> + +<hr /> + +<p><em>Update (August 2017):</em> Gabriel returned to France and left the POPL proceedings at his desk. The proceedings are sitting in the hallway at NEU, in case anyone cares.</p> \ No newline at end of file diff --git a/blog/feeds/dear-diary.atom.xml b/blog/feeds/dear-diary.atom.xml new file mode 100644 index 00000000..bc89d46b --- /dev/null +++ b/blog/feeds/dear-diary.atom.xml @@ -0,0 +1,946 @@ + + + PRL Blog: Posts tagged 'dear diary' + + + urn:http-prl-ccs-neu-edu:-blog-tags-dear-diary-html + 2018-11-24T09:52:58Z + + Disappearing Code + + urn:http-prl-ccs-neu-edu:-blog-2018-11-24-disappearing-code + 2018-11-24T09:52:58Z + 2018-11-24T09:52:58Z + + Ben Greenman + +<p>Two experiences at <a href="https://2018.splashcon.org/home">SPLASH 2018</a> reminded me that software gets thrown away and replaced.</p> +<!-- more--> + +<h3 id="story-1">Story 1</h3> + +<p>The first reminder came near the end of a <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a> by <a href="https://people.csail.mit.edu/rinard/">Martin Rinard</a>. Once upon a time, Martin was working as a consultant and a firm asked him to review a software package. (The firm wanted a second opinion about how the software computed its results.) The firm sent a zipfile; Martin found six versions of the code inside; the firm said &ldquo;well, please check all six versions&rdquo;; and it turned out:</p> + +<ul> + <li><strong>Version 1</strong> : the source code was written in a domain-specific language (DSL) that generated code for the application</li> + <li><strong>Version 2</strong> : the DSL source was the same as version 1, but the generated code was slightly modified</li> + <li>&hellip;</li> + <li><strong>Version 6</strong> : the generated code was the source code and the DSL was gone</li></ul> + +<p>The moral of Martin&rsquo;s story was: (1) the creators of a software system are often different from the maintainers, and (2) researchers need to build tools to help these maintainers.</p> + +<h3 id="story-2">Story 2</h3> + +<p>The second reminder came from a teaching assistant who said the <a href="https://www.cs.cornell.edu/courses/cs3110/2018fa/">functional programming course</a> at their institution was currently using a Python script to test students&rsquo; code. Once upon a time, I was a teaching assistant for the <a href="https://www.cs.cornell.edu/courses/cs3110/2014sp/">same course</a> at the same institution. We had trouble testing students&rsquo; code via the Python script left by the pre&ndash;2013 course staff, so I wrote a <a href="https://gitlab.com/bengreenman/ocaml_tools/">command-line tool</a> to handle the tests and other compile/run/grade tasks. To keep history from repeating itself, I used the same language the course teaches (OCaml) and wrote some documentation &mdash; but it seems like that was not enough. At any rate, writing the tool was a good exercise.</p> + +<blockquote> + <p><em>In the end, everybody must understand for himself.</em> &mdash; <a href="https://dl.acm.org/citation.cfm?id=3731">Per Martin-Löf</a></p></blockquote> + +<h3 id="reflection">Reflection</h3> + +<p>In each story, the maintainers of a software system threw away some old code to make their job easier in the short term. How can we stop this &ldquo;re-inventing the wheel&rdquo; from happening?</p> + +<p>Martin Rinard&rsquo;s solution is to let maintenance programmers keep their current habits, but provide tools to make the short-term, pragmatic solutions into a more robust systems. Search for "<a href="https://people.csail.mit.edu/rinard/paper/osdi04.pdf">failure-oblivious computing</a>" to learn more (this was the topic of his <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a>).</p> + +<p>In Story 1, the maintainers were able to avoid the DSL by modifying an inherited blob of DSL-generated code. If the DSL did not generate code, history might have taken a different course; it might be best to start with a language that offers tools for linguistic re-use, and to build a DSL from these tools &mdash; so there is no generated code. The Racket programming language is exploring this path. For a recent example, see the <a href="https://www2.ccs.neu.edu/racket/pubs/icfp17-acf.pdf">video-lang paper</a>.</p> + +<p>The Story 2 test harness, however, was not generating code. Its maintainers discarded a &ldquo;big&rdquo; program written in a typed functional language in favor of a script. Perhaps we need a language that allows mixing statically-typed and dynamically-typed code (shouts out to <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">my own research</a>).</p> + +<p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p> + +<hr /> + +<h4 id="addendum-comment-from-mitch-wand">Addendum: comment from Mitch Wand</h4> + +<blockquote> + <p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p></blockquote> + +<p>Ermm, this works better for sourdough bread than for people.</p> + +<p>Even in the not-so-real world of checking student solutions, there&rsquo;s often no way of guaranteeing that one half of a pair will be around for the second round. They may be on co-op. Or the course will not be offered the next semster/year/etc. Or the course will change at the next offering (from OCaml to Python or from Racket to Java) so that large chunks of the infrastructure will have to be discarded or rewritten.</p> + +<p>The &ldquo;real&rdquo; solution is to write literate code (as we preached incessantly in PDP), so that the next reader will have at least some clue as about what you wrote. This just may be sufficient incentive to modify rather than rebuild from scratch.</p> + +<p>Ever the optimist, &mdash;Mitch</p> + + Quotes and Stories from "Turing 50" + + urn:http-prl-ccs-neu-edu:-blog-2017-06-24-quotes-and-stories-from-turing-50 + 2017-06-24T20:00:52Z + 2017-06-24T20:00:52Z + + Ben Greenman + +<p>The ACM recently hosted <a href="https://www.acm.org/turing-award-50">a celebration of 50 years of the A.M. Turing award</a>. These are some notes and thoughts from the event, including how Fred Brooks once rented a bus, Don Knuth&rsquo;s outrageous implementation of batch processing, and Judea Pearl&rsquo;s theory of homo sapiens.</p> +<!-- more--> + +<script>document.createElement('dialog');</script> + +<p><strong>Conventions / Disclaimers:</strong></p> + +<ul> + <li> + <p>The blockquotes below are paraphrased, may be incorrect, and may be incorrectly attributed. Make sure to watch the ACM&rsquo;s live stream before quoting anything here!!!</p></li> + <li> + <p>Section-breaks are labeled as &ldquo;panel&rdquo;, &ldquo;talk&rdquo;, &ldquo;question&rdquo;, etc.</p></li> + <li> + <p>This is intentionally &ldquo;bad writing&rdquo; in the Peter Lee sense (see below) &mdash; primarily &ldquo;what I saw&rdquo;, very little about &ldquo;what I thought and felt&rdquo;. A summary in my own words just wouldn&rsquo;t do justice to the panelists.</p></li> + <li> + <p>The &ldquo;Augmented Reality&rdquo; session was my favorite.</p></li></ul> + +<h3 id="opening-remarks">Opening Remarks</h3> + +<h4 id="alan-turing-is-with-us-today"><em>Alan Turing is with us today</em></h4> + +<p>At the start of the event, the <a href="http://users.ecs.soton.ac.uk/wh/">emcee</a> unveiled a bronze bust of Alan Turing. This statue was on display at center stage during the whole event.</p> + +<p>It&rsquo;s a good sculpture and it&rsquo;s good we remember Alan Turing, but I&rsquo;m sad that the ACM would encourage this kind of idol-worship. Let&rsquo;s not forget Turing&rsquo;s excellent teachers and colleagues!</p> + +<h3 id="talk-impact-of-turing-recipients-work">talk: Impact of Turing Recipients&rsquo; Work</h3> + +<h5 id="barbara-liskov">Barbara Liskov</h5> + +<blockquote> + <p><em>the first awards recognized achievements in the standard fields of theory, AI, and systems</em></p> + <p><em>hostile environment around the first awards, trepidation about future awards</em></p> + <p><em>with Unix, Ritchie and Thompson got the design right</em></p> + <p><em>Niklaus Wirth: &ldquo;If I understood how important Pcode was, I would have spent more time designing it&rdquo;</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>What is &ldquo;systems&rdquo; &mdash; does that even have a definition? And Unix is definitely NOT an example of a &ldquo;right design&rdquo;; rather it&rsquo;s a landmark of <a href="https://www.dreamsongs.com/WorseIsBetter.html">worse is better</a> design.</p> + +<h3 id="panel-advances-in-deep-neural-networks">panel: Advances in Deep Neural Networks</h3> + +<h4 id="stuart-russell">Stuart Russell</h4> + +<blockquote> + <p> <em>I work in all areas of AI except for deep learning</em></p></blockquote> + +<h4 id="judea-pearl">Judea Pearl</h4> + +<blockquote> + <p><em>I am a foreigner in this field &hellip; left because human beings are not good at handling information &hellip; people are very good with causal inference, not with statistical inference &hellip; deep learning is statistical</em></p> + <p><em>there is a very old existence proof, homo sapiens took over the planet &hellip; I believe because they had an internal model of their environment &hellip; a drawing of a lion with wings is evidence of this model, you have to have such a model before you can experiment with it and imagine &hellip; snakes have superb optics, result of a long evolution process &hellip; very specific but they cannot build eyeglasses &hellip; humans have an internal model, can build a market based on promises and build large communities based on promises</em></p> + <p><em>I see four levels &hellip; second level is predicting events, if I do X then what? &hellip; third level is counterfactual, if I did things differently then how would the outcome change &hellip; very hard to advance between levels, are we working to help machine learning &lsquo;level up&rsquo;?</em></p> + <p><em>data science is about the relation between data and reality &hellip; data alone is not data science</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>today we can&rsquo;t think without holding a piece of metal</em></p> + <p><em>machine learning is part of computer science rather than AI &hellip; AI is about how to make human &hellip; machine learning is about allocating resources &hellip; matrices are not all of human intelligence &hellip; neural nets are part of a wider toolbox &hellip; too much hype in NLP its just syntax</em></p> + <p><em>huge gap between syntax and semantics &hellip; chat bots are just syntax, don&rsquo;t learn &hellip; faking intelligence with neural nets, so well that you can build a company &hellip;</em></p> + <p><em>real metric is task completion</em></p> + <p><em>if I say &lsquo;a GLEEB walked across the airport&rsquo; then true intelligence can make a lot of educated guesses about a &lsquo;GLEEB&rsquo; without any other context</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>I disagree, ML is part of AI &hellip; understanding intelligence and making intelligent methods for solving AI problems</em></p> + <p><em>to quote Churchhill &lsquo;its not beginning of end, not end, not beginning of end, probably end of beginning&rsquo;</em></p> + <p><em>todays AI powered by hardware and data</em></p> + <p><em>AI cannot yet find our keys</em></p> + <p><em>quote: &lsquo;todays AI is making a perfect chess move while the world is on fire&rsquo; &hellip; ignores context</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>Turing &hellip; a program is a mathematical object &hellip; math community did not recognize this</em></p> + <p><em>lots of grad student descent &hellip; tuning to get performance &hellip; deep learning is neglecting the problem of exponential data &hellip; deep learning is just circuits, circuits lack expressive power &hellip; a human can process data from CERN but a neural net cannot, need to know physics</em></p> + <p><em>probabilistic programming, somewhat under the radar, maybe on the right track &hellip; 10-line program running/generating/updating a large network of possibilities &hellip; more composable and flexible</em></p></blockquote> + +<h5 id="ilya-sutskever">Ilya Sutskever</h5> + +<blockquote> + <p><em>why I like deep learning &hellip; philosophically satisfying &hellip; the hypothesis class is a circuit &hellip; powerful hypothesis class not too many parameters &hellip; can actually find circuits &hellip; &lsquo;violates all theory&rsquo; &hellip; really amazing &hellip; humans can see and hear pretty fast, even though our neurons are pretty slow, perhaps because we do a massively parallel process that doesn&rsquo;t take many steps &hellip; works well enough to be useful</em></p> + <p><em>models e.g. for vision are very hard to understand &hellip; fight fire with fire &hellip; incomprehensible solution to incomprehensible problem</em></p></blockquote> + +<h5 id="raquel-urtasun">Raquel Urtasun</h5> + +<blockquote> + <p><em>the breakthrough in neural nets is not algorithms &hellip; it is tricks, hardware, and grad students</em></p> + <p><em>with neural nets we forget about modeling, uncertainty, and prior knowledge &hellip; perception is a canonical example</em></p></blockquote> + +<h4 id="question-boundaries">question: boundaries</h4> + +<h5 id="judea-pearl">Judea Pearl:</h5> + +<blockquote> + <p><em>glad to see people in deep learning understand its limitations &hellip; is there a clearer definition of the boundaries? Are you worried about bridging the levels factual/inferential/counterfactural?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>the big problem is decision making under uncertainty</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>cognition is a hard problem</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>do you have a clear idea of the boundaries?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>neural nets use back-propagation &hellip; its non-modular, sad fact &hellip; performance and explainability is the tradeoff &hellip; then again people are non-modular</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>AlphaGo is not deep learning &hellip; basically an improved version of the machines Arthur Samuel made in the late 1950s &hellip; the interesting code is in C++ &hellip; rules of go, next moves, searching future states &hellip; depends on transitive closure</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>can AlphaGo take advice from a human?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>not currently, but that would be a new policy to add to the toolbox &hellip; just as neural nets are one tool within AlphaGo</em></p></blockquote> + +<h5 id="raquel-urtasun">Raquel Urtasun</h5> + +<blockquote> + <p><em>no reason to ask if deep learning is going to solve all problems</em></p></blockquote> + +<h4 id="question-education">question: education?</h4> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>indeed, what DO you teach in your neural networks classes?</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>&hellip; chain rule, Taylor expansion</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>teaching is communicating truths &hellip; what is true about neural nets? what are some things that will definitely not happen?</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>Peter Norvig and I have a problem with our AI book &hellip; chapter on vision, chapter on speech, will probably post just point to the neural nets chapter &hellip; we don&rsquo;t really understand! &hellip; really selling students short</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>in labs we talk about what we cannot do &hellip; we all have open problems</em></p> + <p><em>Stuart I hope you have a very good author for the chapters. There are so many open problems to communicate to students!</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>CS cirriculum needs more statistics, inferential thinking &hellip; revise the whole cirriculum bottom-up to weave this in</em></p></blockquote> + +<h4 id="question-could-a-neural-net-fix-my-phone-without-breaking-it">question: could a neural net fix my phone without breaking it?</h4> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>right! big problem that neural nets have no internal model to manipulate</em></p></blockquote> + +<h4 id="question-generalizability">question: generalizability?</h4> + +<h5 id="ilya-sutskever">Ilya Sutskever</h5> + +<blockquote> + <p><em>special-purpose vs. general purpose solution depends on the problem &hellip; most things we give special-purpose solutions &hellip; I guess if you wanted to automate a mathematician that would need to be general</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>always argue with your self &hellip; try to break what you&rsquo;ve built &hellip; there&rsquo;s a system that plays video games just using the pixels on screen as hints &hellip; it&rsquo;s very good at mazes; if a newborn baby learned to play maze games in 2 hours that would be amazing! &hellip; does the system scale? absolutely not</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>When Michael Jordan said &ldquo;people are non-modular&rdquo;, I think he means that people are able to break abstraction barriers when needed.</p> + +<h3 id="panel-restoring-personal-privacy-without-compromising-national-security">panel: Restoring Personal Privacy without Compromising National Security</h3> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>&hellip; wikileaks &hellip; russian hackers &hellip; social emergency &hellip;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>everything I say today is copyleft</em></p> + <p><em>its a misunderstanding to talk about a conflict between security and privacy &hellip; two aspects &hellip; problem goes back to feudalism &hellip; the right to build a castle was granted by the king &hellip; on one hand a castle improves national security &hellip; on the other hand a castle can be used to attack the king &hellip; technology is upsetting the basic notion of private vs. public security &hellip; governments cannot protect citizens and cannot protect themselves &hellip; extremely difficult to prove that a small process is secure</em></p> + <p><em>exceptional access makes it more complex</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>major concern are national security threats and ability of authorities to confound threats &hellip; analogy to printing press &hellip; proclimation of 1635 that only state messengers can carry letters &hellip; 1663 treatise by the national censor, no printing house can have a back door &hellip; the general topic is very old &hellip; title of this session isn&rsquo;t very good, the real dilemma is investigation vs privacy</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>code is law for better or worse, tech is not a tool like a watch &hellip; tech can monitor us and decide when it works &hellip; tech is government, not obedient tools &hellip; the mind is a warrant-proof space &hellip; 5th amendment rights should extend to wearables</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>cannot divorce the security/privacy issues from the current political context &hellip; the serious vulnerabilities are not in math &hellip; they are in users and implementors</em></p></blockquote> + +<h4 id="question-back-doors">question: back doors</h4> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>perhaps we should explain what a back door is</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>agency keeps a master key in escrow</em></p> + <p><em>non-lawyers can and should take a stand on basic issues</em></p> + <p><em>there are legitimate warrant-proof spaces &hellip; electronic extensions of the mind need to be recognized as warrant-proof spaces</em></p> + <p><em>the set of authorities with backdoor access should change as I travel between countries &hellip; but this will lead to a global race to the bottom</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>germany has a law against sex tourism (committed by German citizens visiting other countries) &hellip; neither government will be willing to lose backdoor access</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>technical reasons against backdoors &hellip; (1) &lsquo;weak crypto&rsquo; was implemented, nobody turned it off, is now breakable by anyone in 2015 &hellip; (2) Juniper used non-default crypto parameters, someone (inside?) changed the parameters &hellip; (3) attackers exploit back doors</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>quote &lsquo;you can put a man on the moon, surely you can put a man on the sun&rsquo;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>trouble is getting him back safely</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>I think back doors are okay, but not for personal devices &hellip; need public lab and transparent processes, need separation of powers &hellip; prosecutors are getting cases thrown out because courts do not accept their backdoors &hellip; there is a place for transparent back door tools</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>politicians are rarely technical people</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>tech is not a set of policy-neutral tools, need to address gap of understanding</em></p></blockquote> + +<h4 id="question-">question: ???</h4> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>we don&rsquo;t know how to build good crypto programs &hellip; opponents are debugging our programs with different goals &hellip; we&rsquo;re trying for-all-paths safety (universal) &hellip; they&rsquo;re trying exists-bad-path (existential)</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>cybersecurity market is a lemon market</em></p></blockquote> + +<h4 id="question-how-to-advise">question: how to advise</h4> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>question from audience &lsquo;I am an advisor to a company working with nuclear energy, they are terrified of being attacked, how should I advise them?&rsquo;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>a network like that is probably separated enough to be safe &hellip; the problem is being safe AND connected to the web</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>because the internet of things</em></p></blockquote> + +<h4 id="question-what-should-the-acm-do">question: what should the ACM do?</h4> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>maybe we need increased regulation, the ACM could help bring experts together</em></p></blockquote> + +<h4 id="question-what-is-true-security">question: what is true security</h4> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>it&rsquo;s all the same thing &hellip; gets labeled differently &hellip; just trying to control which bits can go where and who gets to read them</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>security is the absense of being violated</em></p></blockquote> + +<h5 id="paul-syverson-no-true--security-need-to-consider-context">Paul Syverson: <em>no true &gt; security, need to consider context</em></h5> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>problem of our community, have strict standards, may be unrealistic &hellip; maybe a lot more tolerance in practice than our model accepts</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>security and privacy are environmental problems</em></p></blockquote> + +<h4 id="question-can-we-stop-the-needle-in-haystack-search-for-vulnerabilities">question: can we stop the needle-in-haystack search for vulnerabilities?</h4> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>need to build in security from the start</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>need rule of law, transparency, separation of powers</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>stop delaying, instead of spending $$$ on fixing problems, we should invest in solving the fundamental issues</em></p></blockquote> + +<h3 id="panel-preserving-our-past-for-the-future">panel: Preserving our Past for the Future</h3> + +<p>Note: I was volunteering during this session; quotes are sparse</p> + +<h5 id="mahadev-satyanarayanan">Mahadev Satyanarayanan</h5> + +<blockquote> + <p><em>the running system is the total documentation &hellip; there are too many details for prose to capture</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>running old code has a danger of running old bugs</em></p></blockquote> + +<h5 id="">??:</h5> + +<blockquote> + <p><em>what not to save? &hellip; it&rsquo;s very hard to tell in advance</em></p></blockquote> + +<h5 id="mahadev-satyanarayanan">Mahadev Satyanarayanan:</h5> + +<blockquote> + <p><em>there is no absolute censor in a world with caching</em></p></blockquote> + +<h5 id="brewster-kahle">Brewster Kahle</h5> + +<blockquote> + <p><em>asking UNESCO to solve the problem is unrealistic &hellip; need to empower the fanatics, given them tools to preserve data</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>I totally agree with the &ldquo;empower the fanatics&rdquo; sentiment. Today, because of &ldquo;volunteer librarians&rdquo;, I think we&rsquo;re doing pretty well about preserving the past. Suppose I found an old PowerPoint file. I&rsquo;m sure I could find a way to read it with help from the internet &mdash; either by searching Google, pirating an old version of PowerPoint, or asking online forums. So personally I&rsquo;m not worried about losing data we have currently; I&rsquo;m more worried about the future, the internet becoming &ldquo;less chaotic&rdquo;.</p> + +<p>The panel raised a good question about how to preserve research and encourage reproducibility. A <code>.pdf</code> or <code>.tex</code> document is not enough; a virtual machine is okay. Really I think we need a stronger cultural emphasis on literate programming and a mature library like TeX to help authors store and share their work. <a href="https://thegamma.net/">The Gamma</a> seems on the right track.</p> + +<p>I was surprised that the panel did not discuss search, version control, and the ACM&rsquo;s open-access policy.</p> + +<h3 id="panel-moores-law-is-really-dead-whats-next">panel: Moore&rsquo;s Law is Really Dead: What&rsquo;s Next?</h3> + +<p>Note: I was volunteering during this session</p> + +<h5 id="butler-lampson">Butler Lampson</h5> + +<blockquote> + <p><em>there&rsquo;s plenty of room at the top &hellip; with Moore&rsquo;s Law we got improvements at the bottom of the software stack, everything above got to benefit and it was easy to integrate the changes &hellip; there&rsquo;s lots of opportunities to trim fat in the middle/top of the software stack &hellip; these improvements will be harder to integrate, but there&rsquo;s lots of opportunities</em></p></blockquote> + +<h5 id="margaret-martonosi">Margaret Martonosi</h5> + +<blockquote> + <p><em>By the way, don&rsquo;t believe the brochure that says I&rsquo;m at Google. My affiliation is Princeton, Google and I are just friends.</em></p></blockquote> + +<h5 id="butler-lampson">Butler Lampson</h5> + +<blockquote> + <p><em>important to distinguish approximate vs. precise software &hellip; precise software has a specification and the customer cares about that specification &hellip; approximate software doesn&rsquo;t have a hard spec, just needs to approximately work &hellip; the web is approximate, it doesn&rsquo;t work and it doesn&rsquo;t need to! &hellip; windows is precise, definitely has a spec and users definitely care</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>The recording of this panel should be good; it was very lively, very practical. And the first audience question (by <a href="https://people.eecs.berkeley.edu/~pattrsn/">David Patterson</a>) was &ldquo;an A+ question&rdquo;.</p> + +<p>The panel reminded me of a talk by <a href="http://users.ece.utexas.edu/~patt/">Yale Patt</a> about &ldquo;the end&rdquo; of the Von Neumann architecture. His opinion is future computers will be Von Neumann machines that rely on &ldquo;accelerators&rdquo; like a GPU &mdash; computer organization is not going to change, but will expand to have a bigger toolbox. So sure, Moore&rsquo;s Law is dead, but there are many opportunities to make computers faster at places other than the bottom of the software stack.</p> + +<h3 id="panel-challenges-in-ethics-and-computing">panel: Challenges in Ethics and Computing</h3> + +<p>Note: I was volunteering during this session</p> + +<h5 id="raj-reddy">Raj Reddy</h5> + +<blockquote> + <p><em>there are more slaves in the world currently than there were in the US during the civil war &hellip; here is one way technology could help, by giving everone a device to record their location &hellip; if someone&rsquo;s time and location is constant, they may be held against their will</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>do you believe every problem has a technological solution?</em></p></blockquote> + +<h5 id="noel-sharkey">Noel Sharkey</h5> + +<blockquote> + <p><em>yes the training set may be biased against people similar to me, but I want you to consider my case as an individual</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>a very nice Washington Post article</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>whether to encrypt the back hall</em></p></blockquote> + +<h5 id="raj-reddy">Raj Reddy</h5> + +<blockquote> + <p><em>we can sit here and wring our hands, but nothing will come of it unless it is written in the US constitution</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>I did not enjoy this panel. This is an ACM event, not a United Nations event. An ACM-sponsored panel about social and political problems should look for constructive ways that computer science can address these problems. Raj Reddy tried to give constructive solutions, but the panel seemed more interested in complaining about how hopeless things are.</p> + +<p>The comment by Noel Sharkey about &ldquo;consider me as an individual&rdquo; was something I hadn&rsquo;t thought about. Instead of worrying about biased datasets, let&rsquo;s use technology to collect data on an individual instead of abstracting a person by their race, age, or neighborhood.</p> + +<h3 id="talk-computer-science-as-a-major-body-of-accumulated-knowledge">talk: Computer Science as a Major Body of Accumulated Knowledge</h3> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p> <em>don&rsquo;t applaud me, just read my books</em></p> + <p><em>at the time, computer science was AI, numerical analysis, and programming languages</em></p> + <p><em>a colleague said &lsquo;I will believe that computer science is a science when it has 1000 deep theorems&rsquo; &hellip; I am not sure what a deep theorem is but I think its different from what&rsquo;s proven by deep learning</em></p> + <p><em>great privilege that we can invent the problems we work on &hellip; imagination &hellip; physicists can only guess the size of the sun</em></p> + <p><em>I&rsquo;ve always believed computer science and math are two parallel subjects &hellip; sometimes you hear people wondering if one subsumes the other</em></p> + <p><em>when I won the Turing Award, the prize money was about $1,000,000 less than it is today &hellip; I did get a nice Tiffany bowl that my wife and I use to serve strawberries &hellip; strawberries actually taste better &hellip;</em></p> + <p><em>very fortunate in this field &hellip; I&rsquo;m completely worthless as an economic advisor &hellip; it&rsquo;s a game I&rsquo;ve been able to take advantage of</em></p></blockquote> + +<h4 id="question-how-could-you-offer-to-pay-for-tex-bug-reports">question: how could you offer to pay for TeX bug reports?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>well there were many, many bugs &hellip; I stopped doubling at 32768 &hellip; brought people out of nowhere &hellip; next time I check the bug reports will be 2021 &hellip; someone is queueing the bugs reports &hellip; I believe strongly in batch rather than swap-in/swap-out &hellip; last time I checked reports was 2014 so 2021 will be next</em></p> + <p><em>TeX was a literate program, and it helped that I wrote &lsquo;The Errors of TeX&rsquo; about the first N bugs</em></p></blockquote> + +<h4 id="question-do-you-think-computers-will-become-good-composers-of-music-do-you-see-a-role-for-computer-assisted-proving">question: do you think computers will become good composers of music? do you see a role for computer-assisted proving?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>Yes in part, assisted is the key word &hellip; I have a program running now that I hope will help me prove a theorem</em></p></blockquote> + +<h4 id="question-favorite-algorithm">question: favorite algorithm?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>Tarjan&rsquo;s strong components &hellip; short deep useful</em></p></blockquote> + +<h4 id="question-thoughts-on-ai-computers-taking-over">question: thoughts on AI, computers taking over?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>I get scared when I see Stuart Russell making assumptions based on people acting rationally &hellip; then you look at election results</em></p></blockquote> + +<h4 id="question-if-you-could-start-over-and-do-things-differently-what-would-you-change">question: if you could start over and do things differently, what would you change?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>I would use decimal internally in TeX instead of binary</em></p></blockquote> + +<h4 id="question-how-to-record-history">question: how to record history?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>a video &lsquo;Lets not dumb down the history of CS&rsquo; &hellip; used to be history of algorithms &hellip; trouble with funding &hellip; the history is nothing that a non-CS person could not understand &hellip; the whole field of history changed from internal to external &hellip; historians need to be external to get published in journals &hellip; no CS department supports a historian &hellip; recently read a dissertation about the ALGOL 60 copmiler &hellip; very careful, describes data structures and organization &hellip; this kind of thing is what deserves to be called history</em></p></blockquote> + +<h4 id="question-teachers">question: teachers</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>hardest thing for me is choosing between two hypotheses (1) could teach this to anyone (2) only 2% of the world is geeks &hellip; suppose the second is true then you can&rsquo;t talk about how to teach if the teacher is not in the 2% &hellip;</em></p> + <p><em>the newest issue of CACM has a fun typo, &lsquo;deep earning&rsquo;</em></p></blockquote> + +<h3 id="panel-quantum-computing-far-away-around-the-corner-or-maybe-both-at-the-same-time">panel: Quantum Computing: Far Away? Around the Corner? Or Maybe Both at the Same Time?</h3> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>goal to have a 45&ndash;50 qbit machine &hellip; 1 error per 1000 operations &hellip; to test, run sample algorithm, chart output vs. a classical supercomputer &hellip; got to be a supercomputer to finish the computation in time</em></p></blockquote> + +<h5 id="andrew-yao">Andrew Yao</h5> + +<blockquote> + <p><em>I&rsquo;m a believer &hellip; one suggested benchmark is to factor 1000-digit numbers &hellip; impossible to attain &hellip; need to expore new possibilities, take physics attitute</em></p> + <p><em>CS did well attracting attention to quantum &hellip; science should be more open &hellip; share results between physics chemistry CS &hellip; don&rsquo;t just stick to your specialized conferences</em></p> + <p><em>CS departments reception to quantum is less than satisfactory &hellip; 15 years ago, maybe 4 or 5 universities &hellip; now, maybe 7 or 8 .. China doing much better in this regard</em></p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>not useful to make analogy to anything classical &hellip; universal fault tolerance? or computation in the presence of error &hellip; either would be excellent, still a long way off</em></p> + <p><em>IBM put quantum on the cloud &hellip; picked an instruction set that tries to abstract away &hellip; have been 19 published papers on the behavior of this quantum hardware</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>two paths &hellip; finding algorithms, besides Shor&rsquo;s algorithm &hellip; make quantum computer to realize the algorithms &hellip; finding algorithms is very difficult &hellip; information-processing point-of-view</em></p> + <p><em>error correction still small scale &hellip; can we use entanglement between probes to improve accuracy?</em></p></blockquote> + +<h5 id="umesh-vazirani">Umesh Vazirani</h5> + +<blockquote> + <p>_different goals &hellip; maybe you want perfect Qbits for a perfect Hilbert space &hellip; reality is a noisy space &hellip; short run, how to compute with noise &hellip; how to correct errors &hellip;</p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta:</h5> + +<blockquote> + <p>_those 2 paths are the same to me &hellip; we want larger devices with fidelity</p> + <p><em>lets build hardware see where goes &hellip; exciting prospect, computer scientists will explore what they can do with these erroneous qbits &hellip; that&rsquo;s why IBM has the instruction set open to the community</em></p></blockquote> + +<h4 id="question-why-isnt-adding-10-qbits-only-10x-harder">question: why isn&rsquo;t adding 10 qbits only 10x harder?</h4> + +<h5 id="john-martinis">John Martinis:</h5> + +<blockquote> + <p><em>building infrastructure to scale &hellip; not just grad student code &hellip; we&rsquo;re all good coders using standard industry practices for coding</em></p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>fidelity is hard to achieve</em></p></blockquote> + +<h4 id="question-both-ibm-and-google-use-superconducting-storage">question: both IBM and Google use superconducting storage?</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>superconducting scales &hellip; ion traps harder to scale, but we still watch, keep eye on data</em></p></blockquote> + +<h4 id="question-education">question: education</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>I like talking to engineering colleges &hellip; physics and engineering need to work together</em></p></blockquote> + +<h4 id="question-is-quantum-going-to-change-programing-languages">question: is quantum going to change programing languages?</h4> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>yes very different to handle errors &hellip; current challenge is building an abstraction over the superconducting hardware</em></p></blockquote> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>hoping to first expose hardware, then get a model, eventually a language</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>need to start with more algorithms</em></p></blockquote> + +<h4 id="question-what-would-feynman-do">question: what would Feynman do?</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>experiments!</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>yes he&rsquo;d tell us to keep playing, and play with us</em></p></blockquote> + +<h3 id="panel-augmented-reality-from-gaming-to-cognitive-aids-and-beyond">panel: Augmented Reality: From Gaming to Cognitive Aids and Beyond</h3> + +<p>Peter Lee starts off wearing a headset.</p> + +<h5 id="ivan-sutherland">Ivan Sutherland:</h5> + +<blockquote> + <p><em>I can tell you how VR started. Bell Helicopter company wanted to land at night &hellip; put an infrared camera on the landing site and a display in the cockpit &hellip; to test they used the roof of their building &hellip; one day an observer in a Bell office is watching, though the camera, two men playing catch on the roof &hellip; one player threw the ball at the camera and the observer ducked &hellip; he had identified his position with the camera &hellip; my observation was that you didn&rsquo;t need a camera, could substitute a computer &hellip; the rest is history</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>my goal is to augment people &hellip; <a href="https://en.wikipedia.org/wiki/The_Mother_of_All_Demos">Englebart</a> very inspiring &hellip; ok 2 stories &hellip; (1) a student of mine wanted to help picky eaters &hellip; computer vision for when they tried to hide peas under the plate &hellip; projected colors onto the peas, called them &lsquo;disco peas&rsquo;, kids were distracted enough to get over their aversion &hellip; children and parents got involved, new social interaction &hellip; (2) opera makeup for schoolchildren, virtually getting into character &hellip; teenage boys in the classes got to try makeup for the first time &hellip; singers found it useful for rehearsals</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>I feel socially awkward wearing this headset, but I have some of my slides here &hellip; making a wearable headset requires huge effort &hellip; research prototypes can be uncomfortable &hellip; a product needs to be perfect and its very hard to do perfect &hellip; one goal, give Lowe&rsquo;s VR to demo a virtual kitchen &hellip; Case Western anatomy class used virtual cadaver, great collective experience</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>two stories &hellip; (1) Henry Fuchs 1998, working with breast surgeon, try augmented reality to improve the precision of biopsy probe insertion &hellip; 2 years to a working prototype, hard to track surgeon&rsquo;s eyes, display where probe is, where ultrasound is, provide low latency &hellip; one day trying on live patient, worked 100% perfect probe right on the mark, jubilation &hellip; then the doctor had to tell the patient &lsquo;yes it is really cancer&rsquo; &hellip; (2) a challenge, augmented reality EMT training &hellip; real teams, virtual patient, virtual surround &hellip; track real tools, 8 eyes, 8 images, team needs to interact</em></p></blockquote> + +<h4 id="question-what-are-current-uses-of-augmented-reality">question: what are current uses of augmented reality?</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>the pilot of a jumbo jet typically has 1 hour flight experience before he flies for the first time, but extensive training in a flight simulator</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>the <strong>best</strong> AR</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>once I was in a flight simulator with the chief pilot &hellip; and he turned to me and asked &lsquo;have you ever experienced a slow roll in a 747?&rsquo; &hellip; a slow roll is a twisting motion, a very benign maneuver, constant one-G pressure the plane doesn&rsquo;t know its upside down &hellip; &lsquo;here we go&rsquo; and suddenly the world inverted &hellip; I remarked that it was certainly impressive, but didn&rsquo;t you treat the simulator as a real experience, and never attempted anything you would not do in reality? &hellip; &lsquo;that is true, but I am the chief pilot&rsquo;</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>construction, architecture</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>where&rsquo;s the &lsquo;augmented&rsquo;?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>whether augmented or virtual</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>yes we did my kitchen that way, made my wife sick when she tried it</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>surgical</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>still sounds virtual</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>displays on a car, superimposed directions on the tarmac &hellip; one of the users took a vacation and had to use the old GPS technology &hellip; found it very difficult to go back</em></p></blockquote> + +<h4 id="question-ar-tools-for-developers">question: AR tools for developers?</h4> + +<h5 id="blair-macintyre">Blair MacIntyre</h5> + +<blockquote> + <p><em>can developers write apps for the Microsoft <a href="https://www.microsoft.com/en-us/hololens">Hololens</a>?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>we belive in experience, anything we can do to foster experiences is good</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>faking things &hellip; subtle and important &hellip; I remember using a flight simulator, navigating the runway, and I turned my head to see if my wing was going to clip a plane &hellip; turned and there was nothing there &hellip; emotional shock to leave the simulation, I had been flying for 1 hour</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>pilot training is an early adopter because the cost of real planes is so high, impossible to train for emergency situations</em></p> + <p><em>the ultimate goal, you can sit in a virtual chair &hellip; and if the chair has handcuffs you cannot get up &hellip; a virtual bullet is lethal &hellip; probably impossible because bits don&rsquo;t weigh anything &hellip; you know Ben Franklin invented augmented reality, eyeglasses &hellip; the desire outweighs cost &hellip; I cannot see the audience here, maybe it would be good if I had a headset! but Peter took his off</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>because of my slides I couldn&rsquo;t see the audience, but then without the headset I couldn&rsquo;t see them either</em></p></blockquote> + +<h4 id="question-any-challenges-to-suggest-to-the-audience">question: any challenges to suggest to the audience?</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>if we had holographic transport, we wouldn&rsquo;t need planes!</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>maybe, but you need to give attendees a physical presence &hellip; smell, touch</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>what makes us willing to work together? I had a collaboration with three people &hellip; all in different locations .. communicated with a phone &hellip; worked perfectly, because we had worked in the same location first and knew one another so well &hellip; how to get to that point, where a simulation could be a useful tool &hellip; another good observation by Fred Brooks, given a domain X ask how good does the simulation need to be for X &hellip; Licklider told me, you&rsquo;d need damn good fiction to land someone on the moon, the simulation would need to provide every detail &hellip; for flight simulation the user&rsquo;s imagination can fill some gaps, a pilot can recognize an aircraft carrier from a rougher picture</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>at IBM I once hired buses to bring the Poughkeepsie secretaries to the main office &hellip; the secretaries at the two offices only knew one another from the phone &hellip; this one lunch did so much good &hellip; only $75 to rent a bus</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>how important is it to shake hands, to bump into a table?</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>for this conference, I think the live stream is getting a better experience because the cameras zoom in on us, the panelists &hellip; the audience in the back cannot see us, only a picture of us on the monitors</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p>_one excellent video game, starts in the dark, you hear a voice &hellip; turn around and there&rsquo;s a character sitting on a chair &hellip; if you rearrange your furniture he finds a new seat &hellip;</p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>games are a great example &hellip; Pokemon Go &hellip; Apple jusr released an app toolkit &hellip; need to get those in schools, in the hands of kids who can build with them</em></p></blockquote> + +<h4 id="question-ivan-about-your-ultimate-display-paper-what-has-since-surprised-or-frustrated-you">question: Ivan, about your &lsquo;ultimate display&rsquo; paper, what has since surprised or frustrated you?</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>I wasn&rsquo;t surprised because I never had any expectations &hellip; of course sticks are not real &hellip; no assumptions so no strong feelings</em></p></blockquote> + +<h4 id="question-people-already-distracted-by-cell-phones-how-to-manage-all-this-input">question: people already distracted by cell phones, how to manage all this input?</h4> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>good question, how much data you can present to people &hellip; and then the problem with google glass, your companions don&rsquo;t know what you are looking at &hellip; at least with snapchat glasses, you can trust the device is simpler</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>good writing defines reality, bad writing reports it &hellip; with the printing press, quickly went from 30,000 books to over 13,000,000 &hellip; novels evolved shortly after, a new form of expression</em></p></blockquote> + +<h4 id="question-peter-how-long-do-your-people-wear-the-hololens">question: Peter, how long do your people wear the hololens?</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>hard to say &hellip; but often longer than the battery lasts</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>how long does it last?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>depends what you&rsquo;re doing, 3 hours</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>that&rsquo;s encouraging, we had a 30-minute cutoff because participants had enough</em></p></blockquote> + +<h4 id="question-nausea">question: nausea</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>I get nauseous in our minecraft VR &hellip; but there&rsquo;s a pop-out feature where you keep playing, but the game world is in a TV set instead of around you &hellip; can pop back in when you&rsquo;re feeling better</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>we&rsquo;ve seen about 20% of the population gets nauseous</em></p></blockquote> + +<h5 id="audience-member">audience member</h5> + +<blockquote> + <p><em>Dana Boyd conducted an experiment, found the nausea was worse for wemon</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>oculus makes me feel sick, but the hololens has never given me trouble</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>have models to predict head motion, to keep the VR world steadier</em></p></blockquote> + +<h5 id="blair-macintyre">Blair MacIntyre</h5> + +<blockquote> + <p><em>I remember reading papers that measured framerate &hellip; would be interesting to revisit</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>framerate not important, its the latency that gets you &hellip; one colleague of mine, we call her &lsquo;the canary&rsquo; because she&rsquo;s so sensitive, in fact &hellip;</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>talking about nausea is part of the problem, people experience it more &hellip; every time I talk about it in public my co-workers tell me to stop!</em></p> + <p><em>another cool application, there&rsquo;s a hololens app to give blind people a tour of the Redmond office &hellip; you say a building and it takes you there</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>one challenge, the relative brightness of the real and virtual worlds</em></p></blockquote> + +<h4 id="question-any-last-remarks">question: any last remarks</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p>_I hoped from the beginning that AR would be a teaching tool &hellip; I learned that <code>F = MA</code> not from a book but from a large flywheel in the school&rsquo;s basement &hellip; very substantial inertia &hellip; the greatest value for AR would be to show people things in a way that makes the underlying meaning clear &hellip; what color should the hydrogen atoms in a benzene ring be? the color will be fiction, but the quality of learning will depend on that fiction &hellip; challenge for content makers &hellip; what is the haptic experience of feeling bits?</p></blockquote> + +<h3 id="small-group-session">Small Group Session</h3> + +<p>After the last panel, I got to attend a small group session with other students, Dick Karp, and Don Knuth. It doesn&rsquo;t feel right to summarize or quote from the session, but there&rsquo;s one thing I want to write about.</p> + +<p>During the group session, I said something that I now regret. There was a brief silence as the group changed subjects, and someone suggested that we do a round of introductions. I objected, <em>this will take so long</em>, but in fact the introductions were a very good idea.</p> + +<p>Normally, I don&rsquo;t like introductions because they focus on names, backgrounds, and credentials. I don&rsquo;t care about any of these when I&rsquo;m meeting someone! Rather, I prefer to just talk and by-the-way learn about the other person(s). There&rsquo;s an anaology to double-blind reviewing &mdash; the focus should be content and not credentials.</p> + +<p>These introductions were successful for two reasons. First, they gave everyone in the room a turn to speak, and this seemed to help people join the actual discussion sooner. That was strange to me. I always feel a little nervous the first time I speak up in front of a group, but if I really feel like speaking then I can always get over this little barrier. I guess it&rsquo;s not right to assume the nervousness is &ldquo;little&rdquo; for everyone. Second, the introductions format was &ldquo;say your name and a funny fact&rdquo;. This prompt by itself led to some nice conversation topics:</p> + +<ul> + <li>Could a computer program decide whether a statement was funny or not funny?</li> + <li>What kind of humor works in a classroom? In a textbook?</li> + <li>Would this kind of introduction be acceptable in another era or culture, for instance Victorian England?</li></ul> + +<p>&ldquo;Nice&rdquo; in the sense that everyone could contribute, which was a real challenge. Even the question &ldquo;does anyone have a favorite algorithm?&rdquo; didn&rsquo;t have much success fostering discussion.</p> + +<p>Related: a useful greeting at the event was &ldquo;what SIG are you?&rdquo;. The answer was a good hint about what level of abstraction you two could best communicate at.</p> +<!-- ### Misc.--> +<!-- I noticed that some of the young people who served on panels and also gave--> +<!-- long-and-not-very-insightful answers to questions were later on their laptops--> +<!-- as other panels discussed things. I noticed some of the older people who--> +<!-- served on panels falling asleep during other panels--> \ No newline at end of file diff --git a/blog/feeds/dear-diary.rss.xml b/blog/feeds/dear-diary.rss.xml new file mode 100644 index 00000000..289ae10a --- /dev/null +++ b/blog/feeds/dear-diary.rss.xml @@ -0,0 +1,944 @@ + + + + PRL Blog: Posts tagged 'dear diary' + PRL Blog: Posts tagged 'dear diary' + http://prl.ccs.neu.edu/blog/tags/dear-diary.html + Sat, 24 Nov 2018 09:52:58 UT + Sat, 24 Nov 2018 09:52:58 UT + 1800 + + Disappearing Code + http://prl.ccs.neu.edu/blog/2018/11/24/disappearing-code/?utm_source=dear-diary&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-11-24-disappearing-code + Sat, 24 Nov 2018 09:52:58 UT + Ben Greenman + +<p>Two experiences at <a href="https://2018.splashcon.org/home">SPLASH 2018</a> reminded me that software gets thrown away and replaced.</p> +<!-- more--> + +<h3 id="story-1">Story 1</h3> + +<p>The first reminder came near the end of a <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a> by <a href="https://people.csail.mit.edu/rinard/">Martin Rinard</a>. Once upon a time, Martin was working as a consultant and a firm asked him to review a software package. (The firm wanted a second opinion about how the software computed its results.) The firm sent a zipfile; Martin found six versions of the code inside; the firm said &ldquo;well, please check all six versions&rdquo;; and it turned out:</p> + +<ul> + <li><strong>Version 1</strong> : the source code was written in a domain-specific language (DSL) that generated code for the application</li> + <li><strong>Version 2</strong> : the DSL source was the same as version 1, but the generated code was slightly modified</li> + <li>&hellip;</li> + <li><strong>Version 6</strong> : the generated code was the source code and the DSL was gone</li></ul> + +<p>The moral of Martin&rsquo;s story was: (1) the creators of a software system are often different from the maintainers, and (2) researchers need to build tools to help these maintainers.</p> + +<h3 id="story-2">Story 2</h3> + +<p>The second reminder came from a teaching assistant who said the <a href="https://www.cs.cornell.edu/courses/cs3110/2018fa/">functional programming course</a> at their institution was currently using a Python script to test students&rsquo; code. Once upon a time, I was a teaching assistant for the <a href="https://www.cs.cornell.edu/courses/cs3110/2014sp/">same course</a> at the same institution. We had trouble testing students&rsquo; code via the Python script left by the pre&ndash;2013 course staff, so I wrote a <a href="https://gitlab.com/bengreenman/ocaml_tools/">command-line tool</a> to handle the tests and other compile/run/grade tasks. To keep history from repeating itself, I used the same language the course teaches (OCaml) and wrote some documentation &mdash; but it seems like that was not enough. At any rate, writing the tool was a good exercise.</p> + +<blockquote> + <p><em>In the end, everybody must understand for himself.</em> &mdash; <a href="https://dl.acm.org/citation.cfm?id=3731">Per Martin-Löf</a></p></blockquote> + +<h3 id="reflection">Reflection</h3> + +<p>In each story, the maintainers of a software system threw away some old code to make their job easier in the short term. How can we stop this &ldquo;re-inventing the wheel&rdquo; from happening?</p> + +<p>Martin Rinard&rsquo;s solution is to let maintenance programmers keep their current habits, but provide tools to make the short-term, pragmatic solutions into a more robust systems. Search for "<a href="https://people.csail.mit.edu/rinard/paper/osdi04.pdf">failure-oblivious computing</a>" to learn more (this was the topic of his <a href="https://conf.researchr.org/event/sle-2018/papers-a-new-approach-for-software-correctness-and-reliability">talk</a>).</p> + +<p>In Story 1, the maintainers were able to avoid the DSL by modifying an inherited blob of DSL-generated code. If the DSL did not generate code, history might have taken a different course; it might be best to start with a language that offers tools for linguistic re-use, and to build a DSL from these tools &mdash; so there is no generated code. The Racket programming language is exploring this path. For a recent example, see the <a href="https://www2.ccs.neu.edu/racket/pubs/icfp17-acf.pdf">video-lang paper</a>.</p> + +<p>The Story 2 test harness, however, was not generating code. Its maintainers discarded a &ldquo;big&rdquo; program written in a typed functional language in favor of a script. Perhaps we need a language that allows mixing statically-typed and dynamically-typed code (shouts out to <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">my own research</a>).</p> + +<p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p> + +<hr /> + +<h4 id="addendum-comment-from-mitch-wand">Addendum: comment from Mitch Wand</h4> + +<blockquote> + <p>The best solution is probably to start with a team and keep the culture alive. Always pair program!</p></blockquote> + +<p>Ermm, this works better for sourdough bread than for people.</p> + +<p>Even in the not-so-real world of checking student solutions, there&rsquo;s often no way of guaranteeing that one half of a pair will be around for the second round. They may be on co-op. Or the course will not be offered the next semster/year/etc. Or the course will change at the next offering (from OCaml to Python or from Racket to Java) so that large chunks of the infrastructure will have to be discarded or rewritten.</p> + +<p>The &ldquo;real&rdquo; solution is to write literate code (as we preached incessantly in PDP), so that the next reader will have at least some clue as about what you wrote. This just may be sufficient incentive to modify rather than rebuild from scratch.</p> + +<p>Ever the optimist, &mdash;Mitch</p> + + Quotes and Stories from "Turing 50" + http://prl.ccs.neu.edu/blog/2017/06/24/quotes-and-stories-from-turing-50/?utm_source=dear-diary&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-24-quotes-and-stories-from-turing-50 + Sat, 24 Jun 2017 20:00:52 UT + Ben Greenman + +<p>The ACM recently hosted <a href="https://www.acm.org/turing-award-50">a celebration of 50 years of the A.M. Turing award</a>. These are some notes and thoughts from the event, including how Fred Brooks once rented a bus, Don Knuth&rsquo;s outrageous implementation of batch processing, and Judea Pearl&rsquo;s theory of homo sapiens.</p> +<!-- more--> + +<script>document.createElement('dialog');</script> + +<p><strong>Conventions / Disclaimers:</strong></p> + +<ul> + <li> + <p>The blockquotes below are paraphrased, may be incorrect, and may be incorrectly attributed. Make sure to watch the ACM&rsquo;s live stream before quoting anything here!!!</p></li> + <li> + <p>Section-breaks are labeled as &ldquo;panel&rdquo;, &ldquo;talk&rdquo;, &ldquo;question&rdquo;, etc.</p></li> + <li> + <p>This is intentionally &ldquo;bad writing&rdquo; in the Peter Lee sense (see below) &mdash; primarily &ldquo;what I saw&rdquo;, very little about &ldquo;what I thought and felt&rdquo;. A summary in my own words just wouldn&rsquo;t do justice to the panelists.</p></li> + <li> + <p>The &ldquo;Augmented Reality&rdquo; session was my favorite.</p></li></ul> + +<h3 id="opening-remarks">Opening Remarks</h3> + +<h4 id="alan-turing-is-with-us-today"><em>Alan Turing is with us today</em></h4> + +<p>At the start of the event, the <a href="http://users.ecs.soton.ac.uk/wh/">emcee</a> unveiled a bronze bust of Alan Turing. This statue was on display at center stage during the whole event.</p> + +<p>It&rsquo;s a good sculpture and it&rsquo;s good we remember Alan Turing, but I&rsquo;m sad that the ACM would encourage this kind of idol-worship. Let&rsquo;s not forget Turing&rsquo;s excellent teachers and colleagues!</p> + +<h3 id="talk-impact-of-turing-recipients-work">talk: Impact of Turing Recipients&rsquo; Work</h3> + +<h5 id="barbara-liskov">Barbara Liskov</h5> + +<blockquote> + <p><em>the first awards recognized achievements in the standard fields of theory, AI, and systems</em></p> + <p><em>hostile environment around the first awards, trepidation about future awards</em></p> + <p><em>with Unix, Ritchie and Thompson got the design right</em></p> + <p><em>Niklaus Wirth: &ldquo;If I understood how important Pcode was, I would have spent more time designing it&rdquo;</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>What is &ldquo;systems&rdquo; &mdash; does that even have a definition? And Unix is definitely NOT an example of a &ldquo;right design&rdquo;; rather it&rsquo;s a landmark of <a href="https://www.dreamsongs.com/WorseIsBetter.html">worse is better</a> design.</p> + +<h3 id="panel-advances-in-deep-neural-networks">panel: Advances in Deep Neural Networks</h3> + +<h4 id="stuart-russell">Stuart Russell</h4> + +<blockquote> + <p> <em>I work in all areas of AI except for deep learning</em></p></blockquote> + +<h4 id="judea-pearl">Judea Pearl</h4> + +<blockquote> + <p><em>I am a foreigner in this field &hellip; left because human beings are not good at handling information &hellip; people are very good with causal inference, not with statistical inference &hellip; deep learning is statistical</em></p> + <p><em>there is a very old existence proof, homo sapiens took over the planet &hellip; I believe because they had an internal model of their environment &hellip; a drawing of a lion with wings is evidence of this model, you have to have such a model before you can experiment with it and imagine &hellip; snakes have superb optics, result of a long evolution process &hellip; very specific but they cannot build eyeglasses &hellip; humans have an internal model, can build a market based on promises and build large communities based on promises</em></p> + <p><em>I see four levels &hellip; second level is predicting events, if I do X then what? &hellip; third level is counterfactual, if I did things differently then how would the outcome change &hellip; very hard to advance between levels, are we working to help machine learning &lsquo;level up&rsquo;?</em></p> + <p><em>data science is about the relation between data and reality &hellip; data alone is not data science</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>today we can&rsquo;t think without holding a piece of metal</em></p> + <p><em>machine learning is part of computer science rather than AI &hellip; AI is about how to make human &hellip; machine learning is about allocating resources &hellip; matrices are not all of human intelligence &hellip; neural nets are part of a wider toolbox &hellip; too much hype in NLP its just syntax</em></p> + <p><em>huge gap between syntax and semantics &hellip; chat bots are just syntax, don&rsquo;t learn &hellip; faking intelligence with neural nets, so well that you can build a company &hellip;</em></p> + <p><em>real metric is task completion</em></p> + <p><em>if I say &lsquo;a GLEEB walked across the airport&rsquo; then true intelligence can make a lot of educated guesses about a &lsquo;GLEEB&rsquo; without any other context</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>I disagree, ML is part of AI &hellip; understanding intelligence and making intelligent methods for solving AI problems</em></p> + <p><em>to quote Churchhill &lsquo;its not beginning of end, not end, not beginning of end, probably end of beginning&rsquo;</em></p> + <p><em>todays AI powered by hardware and data</em></p> + <p><em>AI cannot yet find our keys</em></p> + <p><em>quote: &lsquo;todays AI is making a perfect chess move while the world is on fire&rsquo; &hellip; ignores context</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>Turing &hellip; a program is a mathematical object &hellip; math community did not recognize this</em></p> + <p><em>lots of grad student descent &hellip; tuning to get performance &hellip; deep learning is neglecting the problem of exponential data &hellip; deep learning is just circuits, circuits lack expressive power &hellip; a human can process data from CERN but a neural net cannot, need to know physics</em></p> + <p><em>probabilistic programming, somewhat under the radar, maybe on the right track &hellip; 10-line program running/generating/updating a large network of possibilities &hellip; more composable and flexible</em></p></blockquote> + +<h5 id="ilya-sutskever">Ilya Sutskever</h5> + +<blockquote> + <p><em>why I like deep learning &hellip; philosophically satisfying &hellip; the hypothesis class is a circuit &hellip; powerful hypothesis class not too many parameters &hellip; can actually find circuits &hellip; &lsquo;violates all theory&rsquo; &hellip; really amazing &hellip; humans can see and hear pretty fast, even though our neurons are pretty slow, perhaps because we do a massively parallel process that doesn&rsquo;t take many steps &hellip; works well enough to be useful</em></p> + <p><em>models e.g. for vision are very hard to understand &hellip; fight fire with fire &hellip; incomprehensible solution to incomprehensible problem</em></p></blockquote> + +<h5 id="raquel-urtasun">Raquel Urtasun</h5> + +<blockquote> + <p><em>the breakthrough in neural nets is not algorithms &hellip; it is tricks, hardware, and grad students</em></p> + <p><em>with neural nets we forget about modeling, uncertainty, and prior knowledge &hellip; perception is a canonical example</em></p></blockquote> + +<h4 id="question-boundaries">question: boundaries</h4> + +<h5 id="judea-pearl">Judea Pearl:</h5> + +<blockquote> + <p><em>glad to see people in deep learning understand its limitations &hellip; is there a clearer definition of the boundaries? Are you worried about bridging the levels factual/inferential/counterfactural?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>the big problem is decision making under uncertainty</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>cognition is a hard problem</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>do you have a clear idea of the boundaries?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>neural nets use back-propagation &hellip; its non-modular, sad fact &hellip; performance and explainability is the tradeoff &hellip; then again people are non-modular</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>AlphaGo is not deep learning &hellip; basically an improved version of the machines Arthur Samuel made in the late 1950s &hellip; the interesting code is in C++ &hellip; rules of go, next moves, searching future states &hellip; depends on transitive closure</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>can AlphaGo take advice from a human?</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>not currently, but that would be a new policy to add to the toolbox &hellip; just as neural nets are one tool within AlphaGo</em></p></blockquote> + +<h5 id="raquel-urtasun">Raquel Urtasun</h5> + +<blockquote> + <p><em>no reason to ask if deep learning is going to solve all problems</em></p></blockquote> + +<h4 id="question-education">question: education?</h4> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>indeed, what DO you teach in your neural networks classes?</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>&hellip; chain rule, Taylor expansion</em></p></blockquote> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>teaching is communicating truths &hellip; what is true about neural nets? what are some things that will definitely not happen?</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>Peter Norvig and I have a problem with our AI book &hellip; chapter on vision, chapter on speech, will probably post just point to the neural nets chapter &hellip; we don&rsquo;t really understand! &hellip; really selling students short</em></p></blockquote> + +<h5 id="fei-fei-li">Fei-Fei Li</h5> + +<blockquote> + <p><em>in labs we talk about what we cannot do &hellip; we all have open problems</em></p> + <p><em>Stuart I hope you have a very good author for the chapters. There are so many open problems to communicate to students!</em></p></blockquote> + +<h5 id="michael-jordan">Michael Jordan</h5> + +<blockquote> + <p><em>CS cirriculum needs more statistics, inferential thinking &hellip; revise the whole cirriculum bottom-up to weave this in</em></p></blockquote> + +<h4 id="question-could-a-neural-net-fix-my-phone-without-breaking-it">question: could a neural net fix my phone without breaking it?</h4> + +<h5 id="judea-pearl">Judea Pearl</h5> + +<blockquote> + <p><em>right! big problem that neural nets have no internal model to manipulate</em></p></blockquote> + +<h4 id="question-generalizability">question: generalizability?</h4> + +<h5 id="ilya-sutskever">Ilya Sutskever</h5> + +<blockquote> + <p><em>special-purpose vs. general purpose solution depends on the problem &hellip; most things we give special-purpose solutions &hellip; I guess if you wanted to automate a mathematician that would need to be general</em></p></blockquote> + +<h5 id="stuart-russell">Stuart Russell</h5> + +<blockquote> + <p><em>always argue with your self &hellip; try to break what you&rsquo;ve built &hellip; there&rsquo;s a system that plays video games just using the pixels on screen as hints &hellip; it&rsquo;s very good at mazes; if a newborn baby learned to play maze games in 2 hours that would be amazing! &hellip; does the system scale? absolutely not</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>When Michael Jordan said &ldquo;people are non-modular&rdquo;, I think he means that people are able to break abstraction barriers when needed.</p> + +<h3 id="panel-restoring-personal-privacy-without-compromising-national-security">panel: Restoring Personal Privacy without Compromising National Security</h3> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>&hellip; wikileaks &hellip; russian hackers &hellip; social emergency &hellip;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>everything I say today is copyleft</em></p> + <p><em>its a misunderstanding to talk about a conflict between security and privacy &hellip; two aspects &hellip; problem goes back to feudalism &hellip; the right to build a castle was granted by the king &hellip; on one hand a castle improves national security &hellip; on the other hand a castle can be used to attack the king &hellip; technology is upsetting the basic notion of private vs. public security &hellip; governments cannot protect citizens and cannot protect themselves &hellip; extremely difficult to prove that a small process is secure</em></p> + <p><em>exceptional access makes it more complex</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>major concern are national security threats and ability of authorities to confound threats &hellip; analogy to printing press &hellip; proclimation of 1635 that only state messengers can carry letters &hellip; 1663 treatise by the national censor, no printing house can have a back door &hellip; the general topic is very old &hellip; title of this session isn&rsquo;t very good, the real dilemma is investigation vs privacy</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>code is law for better or worse, tech is not a tool like a watch &hellip; tech can monitor us and decide when it works &hellip; tech is government, not obedient tools &hellip; the mind is a warrant-proof space &hellip; 5th amendment rights should extend to wearables</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>cannot divorce the security/privacy issues from the current political context &hellip; the serious vulnerabilities are not in math &hellip; they are in users and implementors</em></p></blockquote> + +<h4 id="question-back-doors">question: back doors</h4> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>perhaps we should explain what a back door is</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>agency keeps a master key in escrow</em></p> + <p><em>non-lawyers can and should take a stand on basic issues</em></p> + <p><em>there are legitimate warrant-proof spaces &hellip; electronic extensions of the mind need to be recognized as warrant-proof spaces</em></p> + <p><em>the set of authorities with backdoor access should change as I travel between countries &hellip; but this will lead to a global race to the bottom</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>germany has a law against sex tourism (committed by German citizens visiting other countries) &hellip; neither government will be willing to lose backdoor access</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>technical reasons against backdoors &hellip; (1) &lsquo;weak crypto&rsquo; was implemented, nobody turned it off, is now breakable by anyone in 2015 &hellip; (2) Juniper used non-default crypto parameters, someone (inside?) changed the parameters &hellip; (3) attackers exploit back doors</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>quote &lsquo;you can put a man on the moon, surely you can put a man on the sun&rsquo;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>trouble is getting him back safely</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>I think back doors are okay, but not for personal devices &hellip; need public lab and transparent processes, need separation of powers &hellip; prosecutors are getting cases thrown out because courts do not accept their backdoors &hellip; there is a place for transparent back door tools</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>politicians are rarely technical people</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>tech is not a set of policy-neutral tools, need to address gap of understanding</em></p></blockquote> + +<h4 id="question-">question: ???</h4> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>we don&rsquo;t know how to build good crypto programs &hellip; opponents are debugging our programs with different goals &hellip; we&rsquo;re trying for-all-paths safety (universal) &hellip; they&rsquo;re trying exists-bad-path (existential)</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>cybersecurity market is a lemon market</em></p></blockquote> + +<h4 id="question-how-to-advise">question: how to advise</h4> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>question from audience &lsquo;I am an advisor to a company working with nuclear energy, they are terrified of being attacked, how should I advise them?&rsquo;</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>a network like that is probably separated enough to be safe &hellip; the problem is being safe AND connected to the web</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>because the internet of things</em></p></blockquote> + +<h4 id="question-what-should-the-acm-do">question: what should the ACM do?</h4> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>maybe we need increased regulation, the ACM could help bring experts together</em></p></blockquote> + +<h4 id="question-what-is-true-security">question: what is true security</h4> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>it&rsquo;s all the same thing &hellip; gets labeled differently &hellip; just trying to control which bits can go where and who gets to read them</em></p></blockquote> + +<h5 id="nadia-heninger">Nadia Heninger</h5> + +<blockquote> + <p><em>security is the absense of being violated</em></p></blockquote> + +<h5 id="paul-syverson-no-true--security-need-to-consider-context">Paul Syverson: <em>no true &gt; security, need to consider context</em></h5> + +<h5 id="joan-feigenbaum">Joan Feigenbaum</h5> + +<blockquote> + <p><em>problem of our community, have strict standards, may be unrealistic &hellip; maybe a lot more tolerance in practice than our model accepts</em></p></blockquote> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>security and privacy are environmental problems</em></p></blockquote> + +<h4 id="question-can-we-stop-the-needle-in-haystack-search-for-vulnerabilities">question: can we stop the needle-in-haystack search for vulnerabilities?</h4> + +<h5 id="paul-syverson">Paul Syverson</h5> + +<blockquote> + <p><em>need to build in security from the start</em></p></blockquote> + +<h5 id="bryan-ford">Bryan Ford</h5> + +<blockquote> + <p><em>need rule of law, transparency, separation of powers</em></p></blockquote> + +<h5 id="whitfield-diffie">Whitfield Diffie</h5> + +<blockquote> + <p><em>stop delaying, instead of spending $$$ on fixing problems, we should invest in solving the fundamental issues</em></p></blockquote> + +<h3 id="panel-preserving-our-past-for-the-future">panel: Preserving our Past for the Future</h3> + +<p>Note: I was volunteering during this session; quotes are sparse</p> + +<h5 id="mahadev-satyanarayanan">Mahadev Satyanarayanan</h5> + +<blockquote> + <p><em>the running system is the total documentation &hellip; there are too many details for prose to capture</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>running old code has a danger of running old bugs</em></p></blockquote> + +<h5 id="">??:</h5> + +<blockquote> + <p><em>what not to save? &hellip; it&rsquo;s very hard to tell in advance</em></p></blockquote> + +<h5 id="mahadev-satyanarayanan">Mahadev Satyanarayanan:</h5> + +<blockquote> + <p><em>there is no absolute censor in a world with caching</em></p></blockquote> + +<h5 id="brewster-kahle">Brewster Kahle</h5> + +<blockquote> + <p><em>asking UNESCO to solve the problem is unrealistic &hellip; need to empower the fanatics, given them tools to preserve data</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>I totally agree with the &ldquo;empower the fanatics&rdquo; sentiment. Today, because of &ldquo;volunteer librarians&rdquo;, I think we&rsquo;re doing pretty well about preserving the past. Suppose I found an old PowerPoint file. I&rsquo;m sure I could find a way to read it with help from the internet &mdash; either by searching Google, pirating an old version of PowerPoint, or asking online forums. So personally I&rsquo;m not worried about losing data we have currently; I&rsquo;m more worried about the future, the internet becoming &ldquo;less chaotic&rdquo;.</p> + +<p>The panel raised a good question about how to preserve research and encourage reproducibility. A <code>.pdf</code> or <code>.tex</code> document is not enough; a virtual machine is okay. Really I think we need a stronger cultural emphasis on literate programming and a mature library like TeX to help authors store and share their work. <a href="https://thegamma.net/">The Gamma</a> seems on the right track.</p> + +<p>I was surprised that the panel did not discuss search, version control, and the ACM&rsquo;s open-access policy.</p> + +<h3 id="panel-moores-law-is-really-dead-whats-next">panel: Moore&rsquo;s Law is Really Dead: What&rsquo;s Next?</h3> + +<p>Note: I was volunteering during this session</p> + +<h5 id="butler-lampson">Butler Lampson</h5> + +<blockquote> + <p><em>there&rsquo;s plenty of room at the top &hellip; with Moore&rsquo;s Law we got improvements at the bottom of the software stack, everything above got to benefit and it was easy to integrate the changes &hellip; there&rsquo;s lots of opportunities to trim fat in the middle/top of the software stack &hellip; these improvements will be harder to integrate, but there&rsquo;s lots of opportunities</em></p></blockquote> + +<h5 id="margaret-martonosi">Margaret Martonosi</h5> + +<blockquote> + <p><em>By the way, don&rsquo;t believe the brochure that says I&rsquo;m at Google. My affiliation is Princeton, Google and I are just friends.</em></p></blockquote> + +<h5 id="butler-lampson">Butler Lampson</h5> + +<blockquote> + <p><em>important to distinguish approximate vs. precise software &hellip; precise software has a specification and the customer cares about that specification &hellip; approximate software doesn&rsquo;t have a hard spec, just needs to approximately work &hellip; the web is approximate, it doesn&rsquo;t work and it doesn&rsquo;t need to! &hellip; windows is precise, definitely has a spec and users definitely care</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>The recording of this panel should be good; it was very lively, very practical. And the first audience question (by <a href="https://people.eecs.berkeley.edu/~pattrsn/">David Patterson</a>) was &ldquo;an A+ question&rdquo;.</p> + +<p>The panel reminded me of a talk by <a href="http://users.ece.utexas.edu/~patt/">Yale Patt</a> about &ldquo;the end&rdquo; of the Von Neumann architecture. His opinion is future computers will be Von Neumann machines that rely on &ldquo;accelerators&rdquo; like a GPU &mdash; computer organization is not going to change, but will expand to have a bigger toolbox. So sure, Moore&rsquo;s Law is dead, but there are many opportunities to make computers faster at places other than the bottom of the software stack.</p> + +<h3 id="panel-challenges-in-ethics-and-computing">panel: Challenges in Ethics and Computing</h3> + +<p>Note: I was volunteering during this session</p> + +<h5 id="raj-reddy">Raj Reddy</h5> + +<blockquote> + <p><em>there are more slaves in the world currently than there were in the US during the civil war &hellip; here is one way technology could help, by giving everone a device to record their location &hellip; if someone&rsquo;s time and location is constant, they may be held against their will</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>do you believe every problem has a technological solution?</em></p></blockquote> + +<h5 id="noel-sharkey">Noel Sharkey</h5> + +<blockquote> + <p><em>yes the training set may be biased against people similar to me, but I want you to consider my case as an individual</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>a very nice Washington Post article</em></p></blockquote> + +<h5 id="">??</h5> + +<blockquote> + <p><em>whether to encrypt the back hall</em></p></blockquote> + +<h5 id="raj-reddy">Raj Reddy</h5> + +<blockquote> + <p><em>we can sit here and wring our hands, but nothing will come of it unless it is written in the US constitution</em></p></blockquote> + +<h4 id="thoughts">thoughts</h4> + +<p>I did not enjoy this panel. This is an ACM event, not a United Nations event. An ACM-sponsored panel about social and political problems should look for constructive ways that computer science can address these problems. Raj Reddy tried to give constructive solutions, but the panel seemed more interested in complaining about how hopeless things are.</p> + +<p>The comment by Noel Sharkey about &ldquo;consider me as an individual&rdquo; was something I hadn&rsquo;t thought about. Instead of worrying about biased datasets, let&rsquo;s use technology to collect data on an individual instead of abstracting a person by their race, age, or neighborhood.</p> + +<h3 id="talk-computer-science-as-a-major-body-of-accumulated-knowledge">talk: Computer Science as a Major Body of Accumulated Knowledge</h3> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p> <em>don&rsquo;t applaud me, just read my books</em></p> + <p><em>at the time, computer science was AI, numerical analysis, and programming languages</em></p> + <p><em>a colleague said &lsquo;I will believe that computer science is a science when it has 1000 deep theorems&rsquo; &hellip; I am not sure what a deep theorem is but I think its different from what&rsquo;s proven by deep learning</em></p> + <p><em>great privilege that we can invent the problems we work on &hellip; imagination &hellip; physicists can only guess the size of the sun</em></p> + <p><em>I&rsquo;ve always believed computer science and math are two parallel subjects &hellip; sometimes you hear people wondering if one subsumes the other</em></p> + <p><em>when I won the Turing Award, the prize money was about $1,000,000 less than it is today &hellip; I did get a nice Tiffany bowl that my wife and I use to serve strawberries &hellip; strawberries actually taste better &hellip;</em></p> + <p><em>very fortunate in this field &hellip; I&rsquo;m completely worthless as an economic advisor &hellip; it&rsquo;s a game I&rsquo;ve been able to take advantage of</em></p></blockquote> + +<h4 id="question-how-could-you-offer-to-pay-for-tex-bug-reports">question: how could you offer to pay for TeX bug reports?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>well there were many, many bugs &hellip; I stopped doubling at 32768 &hellip; brought people out of nowhere &hellip; next time I check the bug reports will be 2021 &hellip; someone is queueing the bugs reports &hellip; I believe strongly in batch rather than swap-in/swap-out &hellip; last time I checked reports was 2014 so 2021 will be next</em></p> + <p><em>TeX was a literate program, and it helped that I wrote &lsquo;The Errors of TeX&rsquo; about the first N bugs</em></p></blockquote> + +<h4 id="question-do-you-think-computers-will-become-good-composers-of-music-do-you-see-a-role-for-computer-assisted-proving">question: do you think computers will become good composers of music? do you see a role for computer-assisted proving?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>Yes in part, assisted is the key word &hellip; I have a program running now that I hope will help me prove a theorem</em></p></blockquote> + +<h4 id="question-favorite-algorithm">question: favorite algorithm?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>Tarjan&rsquo;s strong components &hellip; short deep useful</em></p></blockquote> + +<h4 id="question-thoughts-on-ai-computers-taking-over">question: thoughts on AI, computers taking over?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>I get scared when I see Stuart Russell making assumptions based on people acting rationally &hellip; then you look at election results</em></p></blockquote> + +<h4 id="question-if-you-could-start-over-and-do-things-differently-what-would-you-change">question: if you could start over and do things differently, what would you change?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>I would use decimal internally in TeX instead of binary</em></p></blockquote> + +<h4 id="question-how-to-record-history">question: how to record history?</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>a video &lsquo;Lets not dumb down the history of CS&rsquo; &hellip; used to be history of algorithms &hellip; trouble with funding &hellip; the history is nothing that a non-CS person could not understand &hellip; the whole field of history changed from internal to external &hellip; historians need to be external to get published in journals &hellip; no CS department supports a historian &hellip; recently read a dissertation about the ALGOL 60 copmiler &hellip; very careful, describes data structures and organization &hellip; this kind of thing is what deserves to be called history</em></p></blockquote> + +<h4 id="question-teachers">question: teachers</h4> + +<h5 id="donald-knuth">Donald Knuth</h5> + +<blockquote> + <p><em>hardest thing for me is choosing between two hypotheses (1) could teach this to anyone (2) only 2% of the world is geeks &hellip; suppose the second is true then you can&rsquo;t talk about how to teach if the teacher is not in the 2% &hellip;</em></p> + <p><em>the newest issue of CACM has a fun typo, &lsquo;deep earning&rsquo;</em></p></blockquote> + +<h3 id="panel-quantum-computing-far-away-around-the-corner-or-maybe-both-at-the-same-time">panel: Quantum Computing: Far Away? Around the Corner? Or Maybe Both at the Same Time?</h3> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>goal to have a 45&ndash;50 qbit machine &hellip; 1 error per 1000 operations &hellip; to test, run sample algorithm, chart output vs. a classical supercomputer &hellip; got to be a supercomputer to finish the computation in time</em></p></blockquote> + +<h5 id="andrew-yao">Andrew Yao</h5> + +<blockquote> + <p><em>I&rsquo;m a believer &hellip; one suggested benchmark is to factor 1000-digit numbers &hellip; impossible to attain &hellip; need to expore new possibilities, take physics attitute</em></p> + <p><em>CS did well attracting attention to quantum &hellip; science should be more open &hellip; share results between physics chemistry CS &hellip; don&rsquo;t just stick to your specialized conferences</em></p> + <p><em>CS departments reception to quantum is less than satisfactory &hellip; 15 years ago, maybe 4 or 5 universities &hellip; now, maybe 7 or 8 .. China doing much better in this regard</em></p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>not useful to make analogy to anything classical &hellip; universal fault tolerance? or computation in the presence of error &hellip; either would be excellent, still a long way off</em></p> + <p><em>IBM put quantum on the cloud &hellip; picked an instruction set that tries to abstract away &hellip; have been 19 published papers on the behavior of this quantum hardware</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>two paths &hellip; finding algorithms, besides Shor&rsquo;s algorithm &hellip; make quantum computer to realize the algorithms &hellip; finding algorithms is very difficult &hellip; information-processing point-of-view</em></p> + <p><em>error correction still small scale &hellip; can we use entanglement between probes to improve accuracy?</em></p></blockquote> + +<h5 id="umesh-vazirani">Umesh Vazirani</h5> + +<blockquote> + <p>_different goals &hellip; maybe you want perfect Qbits for a perfect Hilbert space &hellip; reality is a noisy space &hellip; short run, how to compute with noise &hellip; how to correct errors &hellip;</p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta:</h5> + +<blockquote> + <p>_those 2 paths are the same to me &hellip; we want larger devices with fidelity</p> + <p><em>lets build hardware see where goes &hellip; exciting prospect, computer scientists will explore what they can do with these erroneous qbits &hellip; that&rsquo;s why IBM has the instruction set open to the community</em></p></blockquote> + +<h4 id="question-why-isnt-adding-10-qbits-only-10x-harder">question: why isn&rsquo;t adding 10 qbits only 10x harder?</h4> + +<h5 id="john-martinis">John Martinis:</h5> + +<blockquote> + <p><em>building infrastructure to scale &hellip; not just grad student code &hellip; we&rsquo;re all good coders using standard industry practices for coding</em></p></blockquote> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>fidelity is hard to achieve</em></p></blockquote> + +<h4 id="question-both-ibm-and-google-use-superconducting-storage">question: both IBM and Google use superconducting storage?</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>superconducting scales &hellip; ion traps harder to scale, but we still watch, keep eye on data</em></p></blockquote> + +<h4 id="question-education">question: education</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>I like talking to engineering colleges &hellip; physics and engineering need to work together</em></p></blockquote> + +<h4 id="question-is-quantum-going-to-change-programing-languages">question: is quantum going to change programing languages?</h4> + +<h5 id="jay-gambetta">Jay Gambetta</h5> + +<blockquote> + <p><em>yes very different to handle errors &hellip; current challenge is building an abstraction over the superconducting hardware</em></p></blockquote> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>hoping to first expose hardware, then get a model, eventually a language</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>need to start with more algorithms</em></p></blockquote> + +<h4 id="question-what-would-feynman-do">question: what would Feynman do?</h4> + +<h5 id="john-martinis">John Martinis</h5> + +<blockquote> + <p><em>experiments!</em></p></blockquote> + +<h5 id="dorit-aharonov">Dorit Aharonov</h5> + +<blockquote> + <p><em>yes he&rsquo;d tell us to keep playing, and play with us</em></p></blockquote> + +<h3 id="panel-augmented-reality-from-gaming-to-cognitive-aids-and-beyond">panel: Augmented Reality: From Gaming to Cognitive Aids and Beyond</h3> + +<p>Peter Lee starts off wearing a headset.</p> + +<h5 id="ivan-sutherland">Ivan Sutherland:</h5> + +<blockquote> + <p><em>I can tell you how VR started. Bell Helicopter company wanted to land at night &hellip; put an infrared camera on the landing site and a display in the cockpit &hellip; to test they used the roof of their building &hellip; one day an observer in a Bell office is watching, though the camera, two men playing catch on the roof &hellip; one player threw the ball at the camera and the observer ducked &hellip; he had identified his position with the camera &hellip; my observation was that you didn&rsquo;t need a camera, could substitute a computer &hellip; the rest is history</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>my goal is to augment people &hellip; <a href="https://en.wikipedia.org/wiki/The_Mother_of_All_Demos">Englebart</a> very inspiring &hellip; ok 2 stories &hellip; (1) a student of mine wanted to help picky eaters &hellip; computer vision for when they tried to hide peas under the plate &hellip; projected colors onto the peas, called them &lsquo;disco peas&rsquo;, kids were distracted enough to get over their aversion &hellip; children and parents got involved, new social interaction &hellip; (2) opera makeup for schoolchildren, virtually getting into character &hellip; teenage boys in the classes got to try makeup for the first time &hellip; singers found it useful for rehearsals</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>I feel socially awkward wearing this headset, but I have some of my slides here &hellip; making a wearable headset requires huge effort &hellip; research prototypes can be uncomfortable &hellip; a product needs to be perfect and its very hard to do perfect &hellip; one goal, give Lowe&rsquo;s VR to demo a virtual kitchen &hellip; Case Western anatomy class used virtual cadaver, great collective experience</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>two stories &hellip; (1) Henry Fuchs 1998, working with breast surgeon, try augmented reality to improve the precision of biopsy probe insertion &hellip; 2 years to a working prototype, hard to track surgeon&rsquo;s eyes, display where probe is, where ultrasound is, provide low latency &hellip; one day trying on live patient, worked 100% perfect probe right on the mark, jubilation &hellip; then the doctor had to tell the patient &lsquo;yes it is really cancer&rsquo; &hellip; (2) a challenge, augmented reality EMT training &hellip; real teams, virtual patient, virtual surround &hellip; track real tools, 8 eyes, 8 images, team needs to interact</em></p></blockquote> + +<h4 id="question-what-are-current-uses-of-augmented-reality">question: what are current uses of augmented reality?</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>the pilot of a jumbo jet typically has 1 hour flight experience before he flies for the first time, but extensive training in a flight simulator</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>the <strong>best</strong> AR</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>once I was in a flight simulator with the chief pilot &hellip; and he turned to me and asked &lsquo;have you ever experienced a slow roll in a 747?&rsquo; &hellip; a slow roll is a twisting motion, a very benign maneuver, constant one-G pressure the plane doesn&rsquo;t know its upside down &hellip; &lsquo;here we go&rsquo; and suddenly the world inverted &hellip; I remarked that it was certainly impressive, but didn&rsquo;t you treat the simulator as a real experience, and never attempted anything you would not do in reality? &hellip; &lsquo;that is true, but I am the chief pilot&rsquo;</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>construction, architecture</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>where&rsquo;s the &lsquo;augmented&rsquo;?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>whether augmented or virtual</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>yes we did my kitchen that way, made my wife sick when she tried it</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>surgical</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>still sounds virtual</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>displays on a car, superimposed directions on the tarmac &hellip; one of the users took a vacation and had to use the old GPS technology &hellip; found it very difficult to go back</em></p></blockquote> + +<h4 id="question-ar-tools-for-developers">question: AR tools for developers?</h4> + +<h5 id="blair-macintyre">Blair MacIntyre</h5> + +<blockquote> + <p><em>can developers write apps for the Microsoft <a href="https://www.microsoft.com/en-us/hololens">Hololens</a>?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>we belive in experience, anything we can do to foster experiences is good</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>faking things &hellip; subtle and important &hellip; I remember using a flight simulator, navigating the runway, and I turned my head to see if my wing was going to clip a plane &hellip; turned and there was nothing there &hellip; emotional shock to leave the simulation, I had been flying for 1 hour</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>pilot training is an early adopter because the cost of real planes is so high, impossible to train for emergency situations</em></p> + <p><em>the ultimate goal, you can sit in a virtual chair &hellip; and if the chair has handcuffs you cannot get up &hellip; a virtual bullet is lethal &hellip; probably impossible because bits don&rsquo;t weigh anything &hellip; you know Ben Franklin invented augmented reality, eyeglasses &hellip; the desire outweighs cost &hellip; I cannot see the audience here, maybe it would be good if I had a headset! but Peter took his off</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>because of my slides I couldn&rsquo;t see the audience, but then without the headset I couldn&rsquo;t see them either</em></p></blockquote> + +<h4 id="question-any-challenges-to-suggest-to-the-audience">question: any challenges to suggest to the audience?</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>if we had holographic transport, we wouldn&rsquo;t need planes!</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>maybe, but you need to give attendees a physical presence &hellip; smell, touch</em></p></blockquote> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>what makes us willing to work together? I had a collaboration with three people &hellip; all in different locations .. communicated with a phone &hellip; worked perfectly, because we had worked in the same location first and knew one another so well &hellip; how to get to that point, where a simulation could be a useful tool &hellip; another good observation by Fred Brooks, given a domain X ask how good does the simulation need to be for X &hellip; Licklider told me, you&rsquo;d need damn good fiction to land someone on the moon, the simulation would need to provide every detail &hellip; for flight simulation the user&rsquo;s imagination can fill some gaps, a pilot can recognize an aircraft carrier from a rougher picture</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>at IBM I once hired buses to bring the Poughkeepsie secretaries to the main office &hellip; the secretaries at the two offices only knew one another from the phone &hellip; this one lunch did so much good &hellip; only $75 to rent a bus</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>how important is it to shake hands, to bump into a table?</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>for this conference, I think the live stream is getting a better experience because the cameras zoom in on us, the panelists &hellip; the audience in the back cannot see us, only a picture of us on the monitors</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p>_one excellent video game, starts in the dark, you hear a voice &hellip; turn around and there&rsquo;s a character sitting on a chair &hellip; if you rearrange your furniture he finds a new seat &hellip;</p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>games are a great example &hellip; Pokemon Go &hellip; Apple jusr released an app toolkit &hellip; need to get those in schools, in the hands of kids who can build with them</em></p></blockquote> + +<h4 id="question-ivan-about-your-ultimate-display-paper-what-has-since-surprised-or-frustrated-you">question: Ivan, about your &lsquo;ultimate display&rsquo; paper, what has since surprised or frustrated you?</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p><em>I wasn&rsquo;t surprised because I never had any expectations &hellip; of course sticks are not real &hellip; no assumptions so no strong feelings</em></p></blockquote> + +<h4 id="question-people-already-distracted-by-cell-phones-how-to-manage-all-this-input">question: people already distracted by cell phones, how to manage all this input?</h4> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>good question, how much data you can present to people &hellip; and then the problem with google glass, your companions don&rsquo;t know what you are looking at &hellip; at least with snapchat glasses, you can trust the device is simpler</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>good writing defines reality, bad writing reports it &hellip; with the printing press, quickly went from 30,000 books to over 13,000,000 &hellip; novels evolved shortly after, a new form of expression</em></p></blockquote> + +<h4 id="question-peter-how-long-do-your-people-wear-the-hololens">question: Peter, how long do your people wear the hololens?</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>hard to say &hellip; but often longer than the battery lasts</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>how long does it last?</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>depends what you&rsquo;re doing, 3 hours</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>that&rsquo;s encouraging, we had a 30-minute cutoff because participants had enough</em></p></blockquote> + +<h4 id="question-nausea">question: nausea</h4> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>I get nauseous in our minecraft VR &hellip; but there&rsquo;s a pop-out feature where you keep playing, but the game world is in a TV set instead of around you &hellip; can pop back in when you&rsquo;re feeling better</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>we&rsquo;ve seen about 20% of the population gets nauseous</em></p></blockquote> + +<h5 id="audience-member">audience member</h5> + +<blockquote> + <p><em>Dana Boyd conducted an experiment, found the nausea was worse for wemon</em></p></blockquote> + +<h5 id="yvonne-rogers">Yvonne Rogers</h5> + +<blockquote> + <p><em>oculus makes me feel sick, but the hololens has never given me trouble</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>have models to predict head motion, to keep the VR world steadier</em></p></blockquote> + +<h5 id="blair-macintyre">Blair MacIntyre</h5> + +<blockquote> + <p><em>I remember reading papers that measured framerate &hellip; would be interesting to revisit</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>framerate not important, its the latency that gets you &hellip; one colleague of mine, we call her &lsquo;the canary&rsquo; because she&rsquo;s so sensitive, in fact &hellip;</em></p></blockquote> + +<h5 id="peter-lee">Peter Lee</h5> + +<blockquote> + <p><em>talking about nausea is part of the problem, people experience it more &hellip; every time I talk about it in public my co-workers tell me to stop!</em></p> + <p><em>another cool application, there&rsquo;s a hololens app to give blind people a tour of the Redmond office &hellip; you say a building and it takes you there</em></p></blockquote> + +<h5 id="fred-brooks">Fred Brooks</h5> + +<blockquote> + <p><em>one challenge, the relative brightness of the real and virtual worlds</em></p></blockquote> + +<h4 id="question-any-last-remarks">question: any last remarks</h4> + +<h5 id="ivan-sutherland">Ivan Sutherland</h5> + +<blockquote> + <p>_I hoped from the beginning that AR would be a teaching tool &hellip; I learned that <code>F = MA</code> not from a book but from a large flywheel in the school&rsquo;s basement &hellip; very substantial inertia &hellip; the greatest value for AR would be to show people things in a way that makes the underlying meaning clear &hellip; what color should the hydrogen atoms in a benzene ring be? the color will be fiction, but the quality of learning will depend on that fiction &hellip; challenge for content makers &hellip; what is the haptic experience of feeling bits?</p></blockquote> + +<h3 id="small-group-session">Small Group Session</h3> + +<p>After the last panel, I got to attend a small group session with other students, Dick Karp, and Don Knuth. It doesn&rsquo;t feel right to summarize or quote from the session, but there&rsquo;s one thing I want to write about.</p> + +<p>During the group session, I said something that I now regret. There was a brief silence as the group changed subjects, and someone suggested that we do a round of introductions. I objected, <em>this will take so long</em>, but in fact the introductions were a very good idea.</p> + +<p>Normally, I don&rsquo;t like introductions because they focus on names, backgrounds, and credentials. I don&rsquo;t care about any of these when I&rsquo;m meeting someone! Rather, I prefer to just talk and by-the-way learn about the other person(s). There&rsquo;s an anaology to double-blind reviewing &mdash; the focus should be content and not credentials.</p> + +<p>These introductions were successful for two reasons. First, they gave everyone in the room a turn to speak, and this seemed to help people join the actual discussion sooner. That was strange to me. I always feel a little nervous the first time I speak up in front of a group, but if I really feel like speaking then I can always get over this little barrier. I guess it&rsquo;s not right to assume the nervousness is &ldquo;little&rdquo; for everyone. Second, the introductions format was &ldquo;say your name and a funny fact&rdquo;. This prompt by itself led to some nice conversation topics:</p> + +<ul> + <li>Could a computer program decide whether a statement was funny or not funny?</li> + <li>What kind of humor works in a classroom? In a textbook?</li> + <li>Would this kind of introduction be acceptable in another era or culture, for instance Victorian England?</li></ul> + +<p>&ldquo;Nice&rdquo; in the sense that everyone could contribute, which was a real challenge. Even the question &ldquo;does anyone have a favorite algorithm?&rdquo; didn&rsquo;t have much success fostering discussion.</p> + +<p>Related: a useful greeting at the event was &ldquo;what SIG are you?&rdquo;. The answer was a good hint about what level of abstraction you two could best communicate at.</p> +<!-- ### Misc.--> +<!-- I noticed that some of the young people who served on panels and also gave--> +<!-- long-and-not-very-insightful answers to questions were later on their laptops--> +<!-- as other panels discussed things. I noticed some of the older people who--> +<!-- served on panels falling asleep during other panels--> \ No newline at end of file diff --git a/blog/feeds/definitions.atom.xml b/blog/feeds/definitions.atom.xml new file mode 100644 index 00000000..47c7df2c --- /dev/null +++ b/blog/feeds/definitions.atom.xml @@ -0,0 +1,166 @@ + + + PRL Blog: Posts tagged 'definitions' + + + urn:http-prl-ccs-neu-edu:-blog-tags-definitions-html + 2019-09-05T10:00:00Z + + Lexical and Dynamic Scope + + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + 2019-09-05T10:00:00Z + 2019-09-05T10:00:00Z + + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> \ No newline at end of file diff --git a/blog/feeds/definitions.rss.xml b/blog/feeds/definitions.rss.xml new file mode 100644 index 00000000..5563782f --- /dev/null +++ b/blog/feeds/definitions.rss.xml @@ -0,0 +1,166 @@ + + + + PRL Blog: Posts tagged 'definitions' + PRL Blog: Posts tagged 'definitions' + http://prl.ccs.neu.edu/blog/tags/definitions.html + Thu, 05 Sep 2019 10:00:00 UT + Thu, 05 Sep 2019 10:00:00 UT + 1800 + + Lexical and Dynamic Scope + http://prl.ccs.neu.edu/blog/2019/09/05/lexical-and-dynamic-scope/?utm_source=definitions&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + Thu, 05 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> \ No newline at end of file diff --git a/blog/feeds/dissertation.atom.xml b/blog/feeds/dissertation.atom.xml new file mode 100644 index 00000000..533b64bf --- /dev/null +++ b/blog/feeds/dissertation.atom.xml @@ -0,0 +1,42 @@ + + + PRL Blog: Posts tagged 'dissertation' + + + urn:http-prl-ccs-neu-edu:-blog-tags-dissertation-html + 2020-12-23T18:21:55Z + + Deep and Shallow Types + + urn:http-prl-ccs-neu-edu:-blog-2020-12-23-deep-and-shallow-types + 2020-12-23T18:21:55Z + 2020-12-23T18:21:55Z + + Ben Greenman + +<p>I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:</p> + +<ul> + <li><a href="http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020</a></li></ul> + +<p>To the PRL: thanks for a wonderful 6.5 years.</p> +<!-- more--> + +<h3 id="abstract">Abstract</h3> + +<blockquote> + <p>The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound&rsquo;&rsquo; mixed-typed languages from others that enforce type obligations locally rather than globally.</p> + <p>In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.</p> + <p>My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.</p> + <p>I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.</p></blockquote> + +<p>Next year, I&rsquo;ll be a <a href="https://cifellows2020.org">CI Fellow</a> at Brown.</p> + + [Conversational Concurrency (cross-post)](https://eighty-twenty.org/2018/01/24/conversational-concurrency) + + urn:http-prl-ccs-neu-edu:-blog-2019-05-11-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency + 2019-05-11T00:03:16Z + 2019-05-11T00:03:16Z + + Tony Garnock-Jones + \ No newline at end of file diff --git a/blog/feeds/dissertation.rss.xml b/blog/feeds/dissertation.rss.xml new file mode 100644 index 00000000..c36722db --- /dev/null +++ b/blog/feeds/dissertation.rss.xml @@ -0,0 +1,40 @@ + + + + PRL Blog: Posts tagged 'dissertation' + PRL Blog: Posts tagged 'dissertation' + http://prl.ccs.neu.edu/blog/tags/dissertation.html + Wed, 23 Dec 2020 18:21:55 UT + Wed, 23 Dec 2020 18:21:55 UT + 1800 + + Deep and Shallow Types + http://prl.ccs.neu.edu/blog/2020/12/23/deep-and-shallow-types/?utm_source=dissertation&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-12-23-deep-and-shallow-types + Wed, 23 Dec 2020 18:21:55 UT + Ben Greenman + +<p>I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:</p> + +<ul> + <li><a href="http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020</a></li></ul> + +<p>To the PRL: thanks for a wonderful 6.5 years.</p> +<!-- more--> + +<h3 id="abstract">Abstract</h3> + +<blockquote> + <p>The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound&rsquo;&rsquo; mixed-typed languages from others that enforce type obligations locally rather than globally.</p> + <p>In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.</p> + <p>My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.</p> + <p>I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.</p></blockquote> + +<p>Next year, I&rsquo;ll be a <a href="https://cifellows2020.org">CI Fellow</a> at Brown.</p> + + [Conversational Concurrency (cross-post)](https://eighty-twenty.org/2018/01/24/conversational-concurrency) + http://prl.ccs.neu.edu/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/?utm_source=dissertation&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-05-11-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency + Sat, 11 May 2019 00:03:16 UT + Tony Garnock-Jones + \ No newline at end of file diff --git a/blog/feeds/dsl.atom.xml b/blog/feeds/dsl.atom.xml new file mode 100644 index 00000000..28068606 --- /dev/null +++ b/blog/feeds/dsl.atom.xml @@ -0,0 +1,733 @@ + + + PRL Blog: Posts tagged 'dsl' + + + urn:http-prl-ccs-neu-edu:-blog-tags-dsl-html + 2018-10-22T15:05:17Z + + Defining Local Bindings in Turnstile Languages + + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + 2018-10-22T15:05:17Z + 2018-10-22T15:05:17Z + + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> \ No newline at end of file diff --git a/blog/feeds/dsl.rss.xml b/blog/feeds/dsl.rss.xml new file mode 100644 index 00000000..e600384a --- /dev/null +++ b/blog/feeds/dsl.rss.xml @@ -0,0 +1,733 @@ + + + + PRL Blog: Posts tagged 'dsl' + PRL Blog: Posts tagged 'dsl' + http://prl.ccs.neu.edu/blog/tags/dsl.html + Mon, 22 Oct 2018 15:05:17 UT + Mon, 22 Oct 2018 15:05:17 UT + 1800 + + Defining Local Bindings in Turnstile Languages + http://prl.ccs.neu.edu/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/?utm_source=dsl&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + Mon, 22 Oct 2018 15:05:17 UT + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> \ No newline at end of file diff --git a/blog/feeds/dynamic-typing.atom.xml b/blog/feeds/dynamic-typing.atom.xml new file mode 100644 index 00000000..0451906d --- /dev/null +++ b/blog/feeds/dynamic-typing.atom.xml @@ -0,0 +1,22 @@ + + + PRL Blog: Posts tagged 'dynamic typing' + + + urn:http-prl-ccs-neu-edu:-blog-tags-dynamic-typing-html + 2017-05-01T12:25:17Z + + Categorical Semantics for Dynamically Typed Programming Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + 2017-05-01T12:25:17Z + 2017-05-01T12:25:17Z + + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> \ No newline at end of file diff --git a/blog/feeds/dynamic-typing.rss.xml b/blog/feeds/dynamic-typing.rss.xml new file mode 100644 index 00000000..cc8a396d --- /dev/null +++ b/blog/feeds/dynamic-typing.rss.xml @@ -0,0 +1,22 @@ + + + + PRL Blog: Posts tagged 'dynamic typing' + PRL Blog: Posts tagged 'dynamic typing' + http://prl.ccs.neu.edu/blog/tags/dynamic-typing.html + Mon, 01 May 2017 12:25:17 UT + Mon, 01 May 2017 12:25:17 UT + 1800 + + Categorical Semantics for Dynamically Typed Programming Languages + http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/?utm_source=dynamic-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + Mon, 01 May 2017 12:25:17 UT + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> \ No newline at end of file diff --git a/blog/feeds/event.atom.xml b/blog/feeds/event.atom.xml new file mode 100644 index 00000000..d31e89cd --- /dev/null +++ b/blog/feeds/event.atom.xml @@ -0,0 +1,178 @@ + + + PRL Blog: Posts tagged 'event' + + + urn:http-prl-ccs-neu-edu:-blog-tags-event-html + 2019-03-09T14:40:16Z + + PLISS: Learn About PL Implementation in a Castle + + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + 2019-03-09T14:40:16Z + 2019-03-09T14:40:16Z + + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> + + The Racket School 2018: Create your own language + + urn:http-prl-ccs-neu-edu:-blog-2018-04-27-the-racket-school-2018-create-your-own-language + 2018-04-27T21:35:22Z + 2018-04-27T21:35:22Z + + Ben Greenman + +<p>The Racket School 2018: Create your own language • 9–13 July • Salt Lake City</p> +<!-- more--> + +<p>The Racket team has spent over thirty years developing and refining a coherent intellectual tradition for studying and building programming languages. This year’s school will introduce participants to Racket’s framework for language-oriented programming, which the summer school faculty recently spelled out in a a cover article in the <a href="https://tinyurl.com/RacketCACM">Communications of the ACM</a></p> + +<p>Concretely, the 2018 Racket Summer School will cover the following topics:</p> + +<ul> + <li>the spectrum of programming languages;</li> + <li>modules and syntax, or languages as libraries;</li> + <li>DrRacket’s support for language-oriented programming;</li> + <li>a domain-specific language for adding types to languages;</li> + <li>tools and techniques for implementing notational conveniences; and</li> + <li>research challenges in language-oriented programming.</li></ul> + +<p>If these topics intrigue you, attend the Racket Summer School:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2018/">http://summer-school.racket-lang.org/2018/</a></li></ul> + +<p>This is not your run-of-the-mill summer school. We will do our best to make it exciting, entertaining, and useful to a broad spectrum of attendees, both academic and industrial.</p> + +<p>P.S. We will send you your first problem set in June, a month before the summer school to whet your appetite.</p> + + Report: PLISS 2017 + + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-report-pliss-2017 + 2017-06-05T15:47:59Z + 2017-06-05T15:47:59Z + + Ming-Ho Yee + +<p>Two weeks ago, I attended the first <a href="https://pliss2017.github.io/">Programming Language Implementation Summer School</a>, held in beautiful Bertinoro, Italy.</p> + +<p>The goal of PLISS was &ldquo;to prepare early graduate students and advanced undergraduates for research in the field,&rdquo; and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.</p> +<!-- more--> + +<h2 id="the-talks">The Talks</h2> + +<p>With the amount of technical content at PLISS, there was easily something for everyone. <a href="http://janvitek.org/">Jan Vitek</a> and <a href="http://tratt.net/laurie/">Laurence Tratt</a> gave lectures that included hands-on exercises where we worked on JITs. <a href="https://www.cs.purdue.edu/homes/suresh/">Suresh Jagannathan</a> dived into the operational semantics of a distributed system, so we could reason about different weak consistency models. Francesco Logozzo gave us a whirlwind tour of abstract interpretation.</p> + +<p>Most of my favorite talks included some form of extra content, such as exercises, live-coding presentations, or demos. I found it really helpful to write actual code and apply what I had just learned, or to look at some concrete examples. The examples and exercises also helped with the pacing, as actively listening to four 90-minute talks every day is exhausting!</p> + +<p>Off the top of my head, these were some of my favorite talks:</p> + +<ul> + <li> + <p><strong>Dynamic Programming Language Implementation with LLVM</strong>, by Petr Maj, Oli Flückiger, and <a href="http://janvitek.org/">Jan Vitek</a>. As the first talk of the summer school, this was a gentle introduction for the rest of the week. We had <a href="https://github.com/PRL-PRG/pliss-rift/">exercises</a> (with intentional bugs to make us think!), and also brief overviews of intermediate languages, static analysis, and garbage collection. These three topics would later show up in more detail.</p></li> + <li> + <p><strong>Micro Virtual Machines</strong>, by <a href="http://users.cecs.anu.edu.au/~steveb/">Steve Blackburn</a>. This talk covered background information on virtual machines, and also the <a href="http://microvm.github.io/">Micro VM</a> project that Steve&rsquo;s group has been working on. A lot of the material was already familiar to me, but I still enjoyed the talk, and even got a few ideas for the project I&rsquo;m working on!</p></li> + <li> + <p><strong>Static Analysis</strong>, by <a href="http://matt.might.net/">Matt Might</a>. Matt&rsquo;s talk was based on one of his <a href="http://matt.might.net/articles/intro-static-analysis/">articles</a> and an older talk he&rsquo;s given. Impressively, the entire example was live-coded, with only a single mistake!</p></li> + <li> + <p><strong>Testing Language Implementations</strong>, by <a href="http://multicore.doc.ic.ac.uk/">Alastair Donaldson</a>. This was an entertaining talk, since Ally showed multiple examples of crashing compilers, and causing other kinds of mischief by triggering compiler bugs.</p></li></ul> + +<p>If you&rsquo;re disappointed that you couldn&rsquo;t see these talks, don&rsquo;t worry! The talks were recorded and will be posted very shortly.</p> + +<h2 id="the-people">The People</h2> + +<p>But there&rsquo;s more to PLISS than the talks. I&rsquo;m referring to <em>networking</em>, or the opportunity to get out and talk to other people about research.</p> + +<p>As an early graduate student, I&rsquo;ve been given a lot of advice about talking to people at conferences and the importance of the &ldquo;hallway track.&rdquo; I still have difficulty doing this at an actual conference, like <a href="http://pldi17.sigplan.org/home">PLDI</a> or <a href="http://2017.ecoop.org/">ECOOP</a>. When there are hundreds of attendees, or when people already know each other and are in conversation groups, I find it difficult to approach them.</p> + +<p>This was not the case at PLISS. There were fewer attendees: about fifty students and a dozen speakers. There was a good mix of undergraduate, master&rsquo;s, first-year PhD, and more senior PhD students. All our breakfasts, lunches, and breaks were together, so we would see the same people again and again, and inevitably start to learn each other&rsquo;s names. The speakers would also be among us, and there was a good ratio of speakers to students for discussions and mealtime mentoring.</p> + +<p>I had many opportunities to practice my &ldquo;research pitch.&rdquo; I talked to senior students and got advice. I talked to junior students and gave advice. Two different people I talked to about my research pointed me to the same paper to read. I found another student who was working with <a href="http://research.cs.wisc.edu/wpis/papers/popl95.pdf">IFDS</a>, an algorithm I have spent much time trying to understand. And, one day at lunch, my table discovered that we were all working on static analysis!</p> + +<p>As much as I enjoyed the talks, I think the best part of PLISS was meeting and talking to other people. You can replace talks with videos (but you lose the speaker-audience interaction), and you can replace conversations with other forms of communication. But there isn&rsquo;t really anything that can replace the serendipity of bumping into someone with a shared interest.</p> + +<h2 id="the-location">The Location</h2> + +<p>Actually, the <em>other</em> best part of PLISS was the location. Italy is a beautiful country with delicious food. And Bertinoro is a small town on the top of a hill, with a breathtaking view of the surrounding area. The lectures were held in a <a href="https://pliss2017.github.io/images/pics/7.jpg">castle at the top of the hill</a> (photo credit: Steve Blackburn). The speakers lived in the castle for the week, while the students lived in the former monastery (seems fitting), which has been renovated into a university residence.</p> + +<p>Here are my two favorite pictures I took (click for full size):</p> + +<p><a href="/img/pliss2017-1.jpg"><img src="/img/pliss2017-1-thumb.jpg" alt="View from the castle" /></a></p> + +<p><a href="/img/pliss2017-2.jpg"><img src="/img/pliss2017-2-thumb.jpg" alt="Panorama" /></a></p> + +<p>Steve Blackburn has more pictures posted on the <a href="https://pliss2017.github.io/">PLISS website</a>.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>PLISS was a wonderful event. Many thanks need to be given to the speakers, organizers, and sponsors, for making this possible!</p> + +<p>If and when there is a second PLISS, I highly encourage students to apply! You will learn a lot from the lectures, from talking to the speakers, and meeting other students. And if it&rsquo;s in Bertinoro again, you can enjoy the weather and nice view!</p> + + PLISS: Oregon without the greek + + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-pliss-oregon-without-the-greek + 2017-02-28T23:01:00Z + 2017-02-28T23:01:00Z + + Jan Vitek + +<p>What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or <a href="https://pliss2017.github.io">PLISS</a> for short) has set out to answer.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2017.github.io/speakers.html">twelve speakers</a> versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.</p> + +<p>Lectures cover current research and future trends in programming language design and implementation, including:</p> + +<ul> + <li>Writing Just-in-time Compilers with LLVM with Jan Vitek</li> + <li>Performance Evaluation and Benchmarking with Laurie Tratt</li> + <li>Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller</li> + <li>High-Performance Fully Concurrent Garbage Collection with Richard Jones</li> + <li>Compiling Dynamic Languages with David Edelsohn</li> + <li>Language-support for Distributed Datastores with Suresh Jagannathan</li> + <li>Testing Programming Language Implementations with Alastair Donaldson</li> + <li>Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might</li> + <li>The evolution of Scala with Martin Odersky</li></ul> + +<p>A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.</p> + +<p>Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.</p> + +<p>More information at:</p> + +<p><a href="https://pliss2017.github.io">https://pliss2017.github.io</a></p> + +<p>(Oregon is a reference to the <a href="https://www.cs.uoregon.edu/research/summerschool/summer17/">OPLSS</a>, in which you may also be interested.)</p> \ No newline at end of file diff --git a/blog/feeds/event.rss.xml b/blog/feeds/event.rss.xml new file mode 100644 index 00000000..e23865cc --- /dev/null +++ b/blog/feeds/event.rss.xml @@ -0,0 +1,172 @@ + + + + PRL Blog: Posts tagged 'event' + PRL Blog: Posts tagged 'event' + http://prl.ccs.neu.edu/blog/tags/event.html + Sat, 09 Mar 2019 14:40:16 UT + Sat, 09 Mar 2019 14:40:16 UT + 1800 + + PLISS: Learn About PL Implementation in a Castle + http://prl.ccs.neu.edu/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/?utm_source=event&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + Sat, 09 Mar 2019 14:40:16 UT + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> + + The Racket School 2018: Create your own language + http://prl.ccs.neu.edu/blog/2018/04/27/the-racket-school-2018-create-your-own-language/?utm_source=event&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-04-27-the-racket-school-2018-create-your-own-language + Fri, 27 Apr 2018 21:35:22 UT + Ben Greenman + +<p>The Racket School 2018: Create your own language • 9–13 July • Salt Lake City</p> +<!-- more--> + +<p>The Racket team has spent over thirty years developing and refining a coherent intellectual tradition for studying and building programming languages. This year’s school will introduce participants to Racket’s framework for language-oriented programming, which the summer school faculty recently spelled out in a a cover article in the <a href="https://tinyurl.com/RacketCACM">Communications of the ACM</a></p> + +<p>Concretely, the 2018 Racket Summer School will cover the following topics:</p> + +<ul> + <li>the spectrum of programming languages;</li> + <li>modules and syntax, or languages as libraries;</li> + <li>DrRacket’s support for language-oriented programming;</li> + <li>a domain-specific language for adding types to languages;</li> + <li>tools and techniques for implementing notational conveniences; and</li> + <li>research challenges in language-oriented programming.</li></ul> + +<p>If these topics intrigue you, attend the Racket Summer School:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2018/">http://summer-school.racket-lang.org/2018/</a></li></ul> + +<p>This is not your run-of-the-mill summer school. We will do our best to make it exciting, entertaining, and useful to a broad spectrum of attendees, both academic and industrial.</p> + +<p>P.S. We will send you your first problem set in June, a month before the summer school to whet your appetite.</p> + + Report: PLISS 2017 + http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/?utm_source=event&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-report-pliss-2017 + Mon, 05 Jun 2017 15:47:59 UT + Ming-Ho Yee + +<p>Two weeks ago, I attended the first <a href="https://pliss2017.github.io/">Programming Language Implementation Summer School</a>, held in beautiful Bertinoro, Italy.</p> + +<p>The goal of PLISS was &ldquo;to prepare early graduate students and advanced undergraduates for research in the field,&rdquo; and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.</p> +<!-- more--> + +<h2 id="the-talks">The Talks</h2> + +<p>With the amount of technical content at PLISS, there was easily something for everyone. <a href="http://janvitek.org/">Jan Vitek</a> and <a href="http://tratt.net/laurie/">Laurence Tratt</a> gave lectures that included hands-on exercises where we worked on JITs. <a href="https://www.cs.purdue.edu/homes/suresh/">Suresh Jagannathan</a> dived into the operational semantics of a distributed system, so we could reason about different weak consistency models. Francesco Logozzo gave us a whirlwind tour of abstract interpretation.</p> + +<p>Most of my favorite talks included some form of extra content, such as exercises, live-coding presentations, or demos. I found it really helpful to write actual code and apply what I had just learned, or to look at some concrete examples. The examples and exercises also helped with the pacing, as actively listening to four 90-minute talks every day is exhausting!</p> + +<p>Off the top of my head, these were some of my favorite talks:</p> + +<ul> + <li> + <p><strong>Dynamic Programming Language Implementation with LLVM</strong>, by Petr Maj, Oli Flückiger, and <a href="http://janvitek.org/">Jan Vitek</a>. As the first talk of the summer school, this was a gentle introduction for the rest of the week. We had <a href="https://github.com/PRL-PRG/pliss-rift/">exercises</a> (with intentional bugs to make us think!), and also brief overviews of intermediate languages, static analysis, and garbage collection. These three topics would later show up in more detail.</p></li> + <li> + <p><strong>Micro Virtual Machines</strong>, by <a href="http://users.cecs.anu.edu.au/~steveb/">Steve Blackburn</a>. This talk covered background information on virtual machines, and also the <a href="http://microvm.github.io/">Micro VM</a> project that Steve&rsquo;s group has been working on. A lot of the material was already familiar to me, but I still enjoyed the talk, and even got a few ideas for the project I&rsquo;m working on!</p></li> + <li> + <p><strong>Static Analysis</strong>, by <a href="http://matt.might.net/">Matt Might</a>. Matt&rsquo;s talk was based on one of his <a href="http://matt.might.net/articles/intro-static-analysis/">articles</a> and an older talk he&rsquo;s given. Impressively, the entire example was live-coded, with only a single mistake!</p></li> + <li> + <p><strong>Testing Language Implementations</strong>, by <a href="http://multicore.doc.ic.ac.uk/">Alastair Donaldson</a>. This was an entertaining talk, since Ally showed multiple examples of crashing compilers, and causing other kinds of mischief by triggering compiler bugs.</p></li></ul> + +<p>If you&rsquo;re disappointed that you couldn&rsquo;t see these talks, don&rsquo;t worry! The talks were recorded and will be posted very shortly.</p> + +<h2 id="the-people">The People</h2> + +<p>But there&rsquo;s more to PLISS than the talks. I&rsquo;m referring to <em>networking</em>, or the opportunity to get out and talk to other people about research.</p> + +<p>As an early graduate student, I&rsquo;ve been given a lot of advice about talking to people at conferences and the importance of the &ldquo;hallway track.&rdquo; I still have difficulty doing this at an actual conference, like <a href="http://pldi17.sigplan.org/home">PLDI</a> or <a href="http://2017.ecoop.org/">ECOOP</a>. When there are hundreds of attendees, or when people already know each other and are in conversation groups, I find it difficult to approach them.</p> + +<p>This was not the case at PLISS. There were fewer attendees: about fifty students and a dozen speakers. There was a good mix of undergraduate, master&rsquo;s, first-year PhD, and more senior PhD students. All our breakfasts, lunches, and breaks were together, so we would see the same people again and again, and inevitably start to learn each other&rsquo;s names. The speakers would also be among us, and there was a good ratio of speakers to students for discussions and mealtime mentoring.</p> + +<p>I had many opportunities to practice my &ldquo;research pitch.&rdquo; I talked to senior students and got advice. I talked to junior students and gave advice. Two different people I talked to about my research pointed me to the same paper to read. I found another student who was working with <a href="http://research.cs.wisc.edu/wpis/papers/popl95.pdf">IFDS</a>, an algorithm I have spent much time trying to understand. And, one day at lunch, my table discovered that we were all working on static analysis!</p> + +<p>As much as I enjoyed the talks, I think the best part of PLISS was meeting and talking to other people. You can replace talks with videos (but you lose the speaker-audience interaction), and you can replace conversations with other forms of communication. But there isn&rsquo;t really anything that can replace the serendipity of bumping into someone with a shared interest.</p> + +<h2 id="the-location">The Location</h2> + +<p>Actually, the <em>other</em> best part of PLISS was the location. Italy is a beautiful country with delicious food. And Bertinoro is a small town on the top of a hill, with a breathtaking view of the surrounding area. The lectures were held in a <a href="https://pliss2017.github.io/images/pics/7.jpg">castle at the top of the hill</a> (photo credit: Steve Blackburn). The speakers lived in the castle for the week, while the students lived in the former monastery (seems fitting), which has been renovated into a university residence.</p> + +<p>Here are my two favorite pictures I took (click for full size):</p> + +<p><a href="/img/pliss2017-1.jpg"><img src="/img/pliss2017-1-thumb.jpg" alt="View from the castle" /></a></p> + +<p><a href="/img/pliss2017-2.jpg"><img src="/img/pliss2017-2-thumb.jpg" alt="Panorama" /></a></p> + +<p>Steve Blackburn has more pictures posted on the <a href="https://pliss2017.github.io/">PLISS website</a>.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>PLISS was a wonderful event. Many thanks need to be given to the speakers, organizers, and sponsors, for making this possible!</p> + +<p>If and when there is a second PLISS, I highly encourage students to apply! You will learn a lot from the lectures, from talking to the speakers, and meeting other students. And if it&rsquo;s in Bertinoro again, you can enjoy the weather and nice view!</p> + + PLISS: Oregon without the greek + http://prl.ccs.neu.edu/blog/2017/02/28/pliss-oregon-without-the-greek/?utm_source=event&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-pliss-oregon-without-the-greek + Tue, 28 Feb 2017 23:01:00 UT + Jan Vitek + +<p>What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or <a href="https://pliss2017.github.io">PLISS</a> for short) has set out to answer.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2017.github.io/speakers.html">twelve speakers</a> versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.</p> + +<p>Lectures cover current research and future trends in programming language design and implementation, including:</p> + +<ul> + <li>Writing Just-in-time Compilers with LLVM with Jan Vitek</li> + <li>Performance Evaluation and Benchmarking with Laurie Tratt</li> + <li>Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller</li> + <li>High-Performance Fully Concurrent Garbage Collection with Richard Jones</li> + <li>Compiling Dynamic Languages with David Edelsohn</li> + <li>Language-support for Distributed Datastores with Suresh Jagannathan</li> + <li>Testing Programming Language Implementations with Alastair Donaldson</li> + <li>Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might</li> + <li>The evolution of Scala with Martin Odersky</li></ul> + +<p>A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.</p> + +<p>Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.</p> + +<p>More information at:</p> + +<p><a href="https://pliss2017.github.io">https://pliss2017.github.io</a></p> + +<p>(Oregon is a reference to the <a href="https://www.cs.uoregon.edu/research/summerschool/summer17/">OPLSS</a>, in which you may also be interested.)</p> \ No newline at end of file diff --git a/blog/feeds/eventual-consistency.atom.xml b/blog/feeds/eventual-consistency.atom.xml new file mode 100644 index 00000000..81280a14 --- /dev/null +++ b/blog/feeds/eventual-consistency.atom.xml @@ -0,0 +1,144 @@ + + + PRL Blog: Posts tagged 'eventual consistency' + + + urn:http-prl-ccs-neu-edu:-blog-tags-eventual-consistency-html + 2017-10-22T11:59:06Z + + Monotonicity Types: Towards A Type System for Eventual Consistency + + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + 2017-10-22T11:59:06Z + 2017-10-22T11:59:06Z + + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> \ No newline at end of file diff --git a/blog/feeds/eventual-consistency.rss.xml b/blog/feeds/eventual-consistency.rss.xml new file mode 100644 index 00000000..2eadc2b9 --- /dev/null +++ b/blog/feeds/eventual-consistency.rss.xml @@ -0,0 +1,144 @@ + + + + PRL Blog: Posts tagged 'eventual consistency' + PRL Blog: Posts tagged 'eventual consistency' + http://prl.ccs.neu.edu/blog/tags/eventual-consistency.html + Sun, 22 Oct 2017 11:59:06 UT + Sun, 22 Oct 2017 11:59:06 UT + 1800 + + Monotonicity Types: Towards A Type System for Eventual Consistency + http://prl.ccs.neu.edu/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/?utm_source=eventual-consistency&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + Sun, 22 Oct 2017 11:59:06 UT + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> \ No newline at end of file diff --git a/blog/feeds/extended-abstract.atom.xml b/blog/feeds/extended-abstract.atom.xml new file mode 100644 index 00000000..7428c4bf --- /dev/null +++ b/blog/feeds/extended-abstract.atom.xml @@ -0,0 +1,247 @@ + + + PRL Blog: Posts tagged 'extended abstract' + + + urn:http-prl-ccs-neu-edu:-blog-tags-extended-abstract-html + 2019-10-31T21:58:26Z + + Complete Monitors for Gradual Types + + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + 2019-10-31T21:58:26Z + 2019-10-31T21:58:26Z + + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + The Behavior of Gradual Types: A User Study + + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + 2018-12-11T19:50:33Z + 2018-12-11T19:50:33Z + + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + A Spectrum of Type Soundness and Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-10-06-a-spectrum-of-type-soundness-and-performance + 2018-10-06T11:23:35Z + 2018-10-06T11:23:35Z + + Ben Greenman + +<p>The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>A Spectrum of Type Soundness and Performance</em> by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018">http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018</a></p></blockquote> + +<p>A dynamically-typed language runs any program that &ldquo;looks good&rdquo; (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both &ldquo;looks good&rdquo; and is well-typed according to a type checker.</p> + +<p>A <em>mixed-typed</em> language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.</p> + +<div class="figure"><img src="/img/mixed-typed-systems-by-year.png" alt="Figure 1: Some mixed-typed languages" /> + <p class="caption">Figure 1: Some mixed-typed languages</p></div> + +<p>These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:</p> + +<ul> + <li><strong>MACLISP</strong> defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the <a href="http://www.softwarepreservation.org/projects/LISP/MIT/Moon-MACLISP_Reference_Manual-Apr_08_1974.pdf">Moonual</a>. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).</li> + <li><strong>Strongtalk</strong> includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a> position paper.</li> + <li><strong>Typed Racket</strong> lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.</li> + <li><strong>Thorn</strong> requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.</li> + <li><strong>Reticulated</strong> lets a program combine static and dynamic <em>expressions</em> and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek <a href="https://dl.acm.org/citation.cfm?id=3009849">POPL 2017</a>).</li></ul> + +<p>That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system&rsquo;s protocol for checking dynamically-typed values at a <em>type boundary</em> (between static and dynamic code).</p> + +<h3 id="main-contribution">Main Contribution</h3> + +<p>In the paper <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>A Spectrum of Type Soundness and Performance</em></a>, we define a tiny mixed-typed language and show three ways to define the behavior of programs &mdash; based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.</p> + +<p>The three behaviors are inspired by existing languages. A <strong>higher order</strong> behavior ensures that dynamically-typed values match the static type at a boundary &mdash; by checking the value when possible, and by monitoring the value&rsquo;s future interactions when necessary. A <strong>first order</strong> behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An <strong>erasure</strong> behavior does no checking whatsoever.</p> + +<blockquote> + <p>Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function <code>f</code>, then one way to enforce the type boundary is to wrap <code>f</code> in a proxy to assert that every future call to <code>f</code> returns a number. In this case, the proxy monitors the behavior of <code>f</code>.</p></blockquote> + +<p>Concretely, the paper defines three formal semantics with the same names. The <strong>higher-order</strong> semantics enforces full types at the boundaries (Section 2.3). The <strong>first-order</strong> semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every &ldquo;selector&rdquo; operation in typed code, e.g., function application, data structure access (Section 2.5). The <strong>erasure</strong> semantics simply ignores the types (Section 2.4).</p> + +<p>Each semantics satisfies a <em>different</em> notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">online supplement</a> gives full proofs.</p> + +<p>The paper has more to say about the meta-theory. See section 2 and section 4.</p> + +<blockquote> + <p>To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for <a href="https://dl.acm.org/citation.cfm?id=2676971">subset of the language</a> (in the spirit of <a href="http://soundiness.org/">soundiness</a>) or use the name &ldquo;type soundness&rdquo; to describe <a href="https://dl.acm.org/citation.cfm?id=2676971">a different property</a>.</p></blockquote> + +<p>Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations &mdash; similar to Typed Racket&rsquo;s <a href="http://docs.racket-lang.org/ts-reference/Typed_Racket_Syntax_Without_Type_Checking.html">no-check</a> language.</p> + +<p>Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (<a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016</a>). The programs we measured are the non-object-oriented ones from our <a href="http://docs.racket-lang.org/gtp-benchmarks/index.html">benchmark suite</a>.</p> + +<p>To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises &mdash; see section 3 of the paper, section B of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, and/or the <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slides</a>.</p> + +<h3 id="implications">Implications</h3> + +<ol> + <li>The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, or <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slide 114</a>.</li> + <li>Programmers using mixed-typed languages need to know what guarantees their types provide. (It is <a href="https://twitter.com/jbandi/status/965005464638541825">not safe to assume that TypeScript types give the same guarantees as OCaml types</a>!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.</li> + <li>The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?</li></ol> + +<blockquote> + <p>Note: in this post, a <em>mixed-typed language</em> is one that allows any combination of static and dynamic typing. A <em>gradually-typed language</em> is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>).</p></blockquote> \ No newline at end of file diff --git a/blog/feeds/extended-abstract.rss.xml b/blog/feeds/extended-abstract.rss.xml new file mode 100644 index 00000000..cedffaf8 --- /dev/null +++ b/blog/feeds/extended-abstract.rss.xml @@ -0,0 +1,243 @@ + + + + PRL Blog: Posts tagged 'extended abstract' + PRL Blog: Posts tagged 'extended abstract' + http://prl.ccs.neu.edu/blog/tags/extended-abstract.html + Thu, 31 Oct 2019 21:58:26 UT + Thu, 31 Oct 2019 21:58:26 UT + 1800 + + Complete Monitors for Gradual Types + http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/?utm_source=extended-abstract&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + Thu, 31 Oct 2019 21:58:26 UT + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + The Behavior of Gradual Types: A User Study + http://prl.ccs.neu.edu/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/?utm_source=extended-abstract&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + Tue, 11 Dec 2018 19:50:33 UT + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + A Spectrum of Type Soundness and Performance + http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/?utm_source=extended-abstract&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-06-a-spectrum-of-type-soundness-and-performance + Sat, 06 Oct 2018 11:23:35 UT + Ben Greenman + +<p>The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>A Spectrum of Type Soundness and Performance</em> by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018">http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018</a></p></blockquote> + +<p>A dynamically-typed language runs any program that &ldquo;looks good&rdquo; (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both &ldquo;looks good&rdquo; and is well-typed according to a type checker.</p> + +<p>A <em>mixed-typed</em> language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.</p> + +<div class="figure"><img src="/img/mixed-typed-systems-by-year.png" alt="Figure 1: Some mixed-typed languages" /> + <p class="caption">Figure 1: Some mixed-typed languages</p></div> + +<p>These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:</p> + +<ul> + <li><strong>MACLISP</strong> defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the <a href="http://www.softwarepreservation.org/projects/LISP/MIT/Moon-MACLISP_Reference_Manual-Apr_08_1974.pdf">Moonual</a>. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).</li> + <li><strong>Strongtalk</strong> includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a> position paper.</li> + <li><strong>Typed Racket</strong> lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.</li> + <li><strong>Thorn</strong> requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.</li> + <li><strong>Reticulated</strong> lets a program combine static and dynamic <em>expressions</em> and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek <a href="https://dl.acm.org/citation.cfm?id=3009849">POPL 2017</a>).</li></ul> + +<p>That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system&rsquo;s protocol for checking dynamically-typed values at a <em>type boundary</em> (between static and dynamic code).</p> + +<h3 id="main-contribution">Main Contribution</h3> + +<p>In the paper <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>A Spectrum of Type Soundness and Performance</em></a>, we define a tiny mixed-typed language and show three ways to define the behavior of programs &mdash; based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.</p> + +<p>The three behaviors are inspired by existing languages. A <strong>higher order</strong> behavior ensures that dynamically-typed values match the static type at a boundary &mdash; by checking the value when possible, and by monitoring the value&rsquo;s future interactions when necessary. A <strong>first order</strong> behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An <strong>erasure</strong> behavior does no checking whatsoever.</p> + +<blockquote> + <p>Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function <code>f</code>, then one way to enforce the type boundary is to wrap <code>f</code> in a proxy to assert that every future call to <code>f</code> returns a number. In this case, the proxy monitors the behavior of <code>f</code>.</p></blockquote> + +<p>Concretely, the paper defines three formal semantics with the same names. The <strong>higher-order</strong> semantics enforces full types at the boundaries (Section 2.3). The <strong>first-order</strong> semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every &ldquo;selector&rdquo; operation in typed code, e.g., function application, data structure access (Section 2.5). The <strong>erasure</strong> semantics simply ignores the types (Section 2.4).</p> + +<p>Each semantics satisfies a <em>different</em> notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">online supplement</a> gives full proofs.</p> + +<p>The paper has more to say about the meta-theory. See section 2 and section 4.</p> + +<blockquote> + <p>To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for <a href="https://dl.acm.org/citation.cfm?id=2676971">subset of the language</a> (in the spirit of <a href="http://soundiness.org/">soundiness</a>) or use the name &ldquo;type soundness&rdquo; to describe <a href="https://dl.acm.org/citation.cfm?id=2676971">a different property</a>.</p></blockquote> + +<p>Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations &mdash; similar to Typed Racket&rsquo;s <a href="http://docs.racket-lang.org/ts-reference/Typed_Racket_Syntax_Without_Type_Checking.html">no-check</a> language.</p> + +<p>Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (<a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016</a>). The programs we measured are the non-object-oriented ones from our <a href="http://docs.racket-lang.org/gtp-benchmarks/index.html">benchmark suite</a>.</p> + +<p>To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises &mdash; see section 3 of the paper, section B of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, and/or the <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slides</a>.</p> + +<h3 id="implications">Implications</h3> + +<ol> + <li>The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, or <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slide 114</a>.</li> + <li>Programmers using mixed-typed languages need to know what guarantees their types provide. (It is <a href="https://twitter.com/jbandi/status/965005464638541825">not safe to assume that TypeScript types give the same guarantees as OCaml types</a>!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.</li> + <li>The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?</li></ol> + +<blockquote> + <p>Note: in this post, a <em>mixed-typed language</em> is one that allows any combination of static and dynamic typing. A <em>gradually-typed language</em> is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>).</p></blockquote> \ No newline at end of file diff --git a/blog/feeds/final-encoding.atom.xml b/blog/feeds/final-encoding.atom.xml new file mode 100644 index 00000000..707e1ee2 --- /dev/null +++ b/blog/feeds/final-encoding.atom.xml @@ -0,0 +1,285 @@ + + + PRL Blog: Posts tagged 'final encoding' + + + urn:http-prl-ccs-neu-edu:-blog-tags-final-encoding-html + 2017-09-27T15:44:57Z + + Final Algebra Semantics is Observational Equivalence + + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + 2017-09-27T15:44:57Z + 2017-09-27T15:44:57Z + + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> \ No newline at end of file diff --git a/blog/feeds/final-encoding.rss.xml b/blog/feeds/final-encoding.rss.xml new file mode 100644 index 00000000..fa3db5cf --- /dev/null +++ b/blog/feeds/final-encoding.rss.xml @@ -0,0 +1,285 @@ + + + + PRL Blog: Posts tagged 'final encoding' + PRL Blog: Posts tagged 'final encoding' + http://prl.ccs.neu.edu/blog/tags/final-encoding.html + Wed, 27 Sep 2017 15:44:57 UT + Wed, 27 Sep 2017 15:44:57 UT + 1800 + + Final Algebra Semantics is Observational Equivalence + http://prl.ccs.neu.edu/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/?utm_source=final-encoding&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + Wed, 27 Sep 2017 15:44:57 UT + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> \ No newline at end of file diff --git a/blog/feeds/galois-connection.atom.xml b/blog/feeds/galois-connection.atom.xml new file mode 100644 index 00000000..835a31df --- /dev/null +++ b/blog/feeds/galois-connection.atom.xml @@ -0,0 +1,85 @@ + + + PRL Blog: Posts tagged 'galois connection' + + + urn:http-prl-ccs-neu-edu:-blog-tags-galois-connection-html + 2016-11-16T00:00:00Z + + Understanding Constructive Galois Connections + + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + 2016-11-16T00:00:00Z + 2016-11-16T00:00:00Z + + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/galois-connection.rss.xml b/blog/feeds/galois-connection.rss.xml new file mode 100644 index 00000000..7bee27b8 --- /dev/null +++ b/blog/feeds/galois-connection.rss.xml @@ -0,0 +1,85 @@ + + + + PRL Blog: Posts tagged 'galois connection' + PRL Blog: Posts tagged 'galois connection' + http://prl.ccs.neu.edu/blog/tags/galois-connection.html + Wed, 16 Nov 2016 00:00:00 UT + Wed, 16 Nov 2016 00:00:00 UT + 1800 + + Understanding Constructive Galois Connections + http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/?utm_source=galois-connection&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + Wed, 16 Nov 2016 00:00:00 UT + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/garbage-collection.atom.xml b/blog/feeds/garbage-collection.atom.xml new file mode 100644 index 00000000..8eaed0ea --- /dev/null +++ b/blog/feeds/garbage-collection.atom.xml @@ -0,0 +1,456 @@ + + + PRL Blog: Posts tagged 'garbage collection' + + + urn:http-prl-ccs-neu-edu:-blog-tags-garbage-collection-html + 2016-05-24T10:51:34Z + + Measuring GC latencies in Haskell, OCaml, Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + 2016-05-24T10:51:34Z + 2016-05-24T10:51:34Z + + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/garbage-collection.rss.xml b/blog/feeds/garbage-collection.rss.xml new file mode 100644 index 00000000..5f49b921 --- /dev/null +++ b/blog/feeds/garbage-collection.rss.xml @@ -0,0 +1,456 @@ + + + + PRL Blog: Posts tagged 'garbage collection' + PRL Blog: Posts tagged 'garbage collection' + http://prl.ccs.neu.edu/blog/tags/garbage-collection.html + Tue, 24 May 2016 10:51:34 UT + Tue, 24 May 2016 10:51:34 UT + 1800 + + Measuring GC latencies in Haskell, OCaml, Racket + http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/?utm_source=garbage-collection&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + Tue, 24 May 2016 10:51:34 UT + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/ghc.atom.xml b/blog/feeds/ghc.atom.xml new file mode 100644 index 00000000..c1c10f66 --- /dev/null +++ b/blog/feeds/ghc.atom.xml @@ -0,0 +1,456 @@ + + + PRL Blog: Posts tagged 'ghc' + + + urn:http-prl-ccs-neu-edu:-blog-tags-ghc-html + 2016-05-24T10:51:34Z + + Measuring GC latencies in Haskell, OCaml, Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + 2016-05-24T10:51:34Z + 2016-05-24T10:51:34Z + + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/ghc.rss.xml b/blog/feeds/ghc.rss.xml new file mode 100644 index 00000000..e142a680 --- /dev/null +++ b/blog/feeds/ghc.rss.xml @@ -0,0 +1,456 @@ + + + + PRL Blog: Posts tagged 'ghc' + PRL Blog: Posts tagged 'ghc' + http://prl.ccs.neu.edu/blog/tags/ghc.html + Tue, 24 May 2016 10:51:34 UT + Tue, 24 May 2016 10:51:34 UT + 1800 + + Measuring GC latencies in Haskell, OCaml, Racket + http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/?utm_source=ghc&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + Tue, 24 May 2016 10:51:34 UT + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/gradual-typing.atom.xml b/blog/feeds/gradual-typing.atom.xml new file mode 100644 index 00000000..830f6701 --- /dev/null +++ b/blog/feeds/gradual-typing.atom.xml @@ -0,0 +1,535 @@ + + + PRL Blog: Posts tagged 'gradual typing' + + + urn:http-prl-ccs-neu-edu:-blog-tags-gradual-typing-html + 2019-10-31T21:58:26Z + + Complete Monitors for Gradual Types + + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + 2019-10-31T21:58:26Z + 2019-10-31T21:58:26Z + + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + The Behavior of Gradual Types: A User Study + + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + 2018-12-11T19:50:33Z + 2018-12-11T19:50:33Z + + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + A Spectrum of Type Soundness and Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-10-06-a-spectrum-of-type-soundness-and-performance + 2018-10-06T11:23:35Z + 2018-10-06T11:23:35Z + + Ben Greenman + +<p>The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>A Spectrum of Type Soundness and Performance</em> by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018">http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018</a></p></blockquote> + +<p>A dynamically-typed language runs any program that &ldquo;looks good&rdquo; (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both &ldquo;looks good&rdquo; and is well-typed according to a type checker.</p> + +<p>A <em>mixed-typed</em> language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.</p> + +<div class="figure"><img src="/img/mixed-typed-systems-by-year.png" alt="Figure 1: Some mixed-typed languages" /> + <p class="caption">Figure 1: Some mixed-typed languages</p></div> + +<p>These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:</p> + +<ul> + <li><strong>MACLISP</strong> defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the <a href="http://www.softwarepreservation.org/projects/LISP/MIT/Moon-MACLISP_Reference_Manual-Apr_08_1974.pdf">Moonual</a>. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).</li> + <li><strong>Strongtalk</strong> includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a> position paper.</li> + <li><strong>Typed Racket</strong> lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.</li> + <li><strong>Thorn</strong> requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.</li> + <li><strong>Reticulated</strong> lets a program combine static and dynamic <em>expressions</em> and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek <a href="https://dl.acm.org/citation.cfm?id=3009849">POPL 2017</a>).</li></ul> + +<p>That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system&rsquo;s protocol for checking dynamically-typed values at a <em>type boundary</em> (between static and dynamic code).</p> + +<h3 id="main-contribution">Main Contribution</h3> + +<p>In the paper <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>A Spectrum of Type Soundness and Performance</em></a>, we define a tiny mixed-typed language and show three ways to define the behavior of programs &mdash; based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.</p> + +<p>The three behaviors are inspired by existing languages. A <strong>higher order</strong> behavior ensures that dynamically-typed values match the static type at a boundary &mdash; by checking the value when possible, and by monitoring the value&rsquo;s future interactions when necessary. A <strong>first order</strong> behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An <strong>erasure</strong> behavior does no checking whatsoever.</p> + +<blockquote> + <p>Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function <code>f</code>, then one way to enforce the type boundary is to wrap <code>f</code> in a proxy to assert that every future call to <code>f</code> returns a number. In this case, the proxy monitors the behavior of <code>f</code>.</p></blockquote> + +<p>Concretely, the paper defines three formal semantics with the same names. The <strong>higher-order</strong> semantics enforces full types at the boundaries (Section 2.3). The <strong>first-order</strong> semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every &ldquo;selector&rdquo; operation in typed code, e.g., function application, data structure access (Section 2.5). The <strong>erasure</strong> semantics simply ignores the types (Section 2.4).</p> + +<p>Each semantics satisfies a <em>different</em> notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">online supplement</a> gives full proofs.</p> + +<p>The paper has more to say about the meta-theory. See section 2 and section 4.</p> + +<blockquote> + <p>To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for <a href="https://dl.acm.org/citation.cfm?id=2676971">subset of the language</a> (in the spirit of <a href="http://soundiness.org/">soundiness</a>) or use the name &ldquo;type soundness&rdquo; to describe <a href="https://dl.acm.org/citation.cfm?id=2676971">a different property</a>.</p></blockquote> + +<p>Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations &mdash; similar to Typed Racket&rsquo;s <a href="http://docs.racket-lang.org/ts-reference/Typed_Racket_Syntax_Without_Type_Checking.html">no-check</a> language.</p> + +<p>Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (<a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016</a>). The programs we measured are the non-object-oriented ones from our <a href="http://docs.racket-lang.org/gtp-benchmarks/index.html">benchmark suite</a>.</p> + +<p>To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises &mdash; see section 3 of the paper, section B of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, and/or the <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slides</a>.</p> + +<h3 id="implications">Implications</h3> + +<ol> + <li>The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, or <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slide 114</a>.</li> + <li>Programmers using mixed-typed languages need to know what guarantees their types provide. (It is <a href="https://twitter.com/jbandi/status/965005464638541825">not safe to assume that TypeScript types give the same guarantees as OCaml types</a>!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.</li> + <li>The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?</li></ol> + +<blockquote> + <p>Note: in this post, a <em>mixed-typed language</em> is one that allows any combination of static and dynamic typing. A <em>gradually-typed language</em> is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>).</p></blockquote> + + Sampling Gradual Typing Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + 2018-05-08T15:37:37Z + 2018-05-08T15:37:37Z + Ben Greenman + Zeina Migeed + + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> + + Gradual Typing Across the Spectrum, part II + + urn:http-prl-ccs-neu-edu:-blog-2017-08-22-gradual-typing-across-the-spectrum-part-ii + 2017-08-22T15:54:06Z + 2017-08-22T15:54:06Z + + Ben Greenman + +<p>Last week, Northeastern hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:</p> + +<p><a href="http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html">http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html</a></p> + +<p>A common thread among the talks was the question: <em>how to convert a research idea into a tool for software developers?</em></p> +<!-- more--> + +<p>In my mind, gradual typing <em>is</em> an answer to one instance of this question. The research idea is strong static type systems, and the software developers are the millions using dynamically typed languages. I know that static typing can make programs easier to write and maintain. The developers know that dynamic typing has benefits; moreover they know better than to migrate their code from one language to another on a whim. Gradual typing is a linguistic solution to the problem of <em>adding</em> the benefits of static typing to a dynamically typed language.</p> + +<p>Enough opinions, let&rsquo;s talk about the talks.</p> + +<p>The morning session consisted of four talks:</p> + +<ul> + <li> + <p><a href="https://www.cs.umd.edu/people/milod">Milod Kazerounian</a> (<a href="https://www.cs.umd.edu/">UMD</a>) spoke about upgrading the <a href="https://github.com/plum-umd/rdl">RDL</a> type checker for Ruby with support for refinement types. The idea is to compile Ruby code and types to <a href="https://emina.github.io/rosette/">Rosette</a>, and profit from <a href="http://yices.csl.sri.com/papers/cav2007.pdf">SMT</a>-assisted type checking.</p></li> + <li> + <p><a href="http://ambrosebs.com/">Ambrose Bonnaire-Sergeant</a> (<a href="https://www.cs.indiana.edu/">IU</a>, <a href="http://ambrosebs.com/talks/squash-work-boston-pi-2017.pdf">slides</a>) has been inferring <em>useful</em> <a href="http://typedclojure.org/">Typed Clojure</a> types through dynamic analysis of Clojure programs. His tool observes how values flow through a program at run-time, then lifts these observations into possibly-recursive, possibly-incorrect type annotations. The surprising result is that the tool quickly (1&ndash;2 seconds per unit test, I think) infers types that can help a developer start annotating a program.</p></li> + <li> + <p><a href="http://ccs.neu.edu/~types/">Ben Greenman</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>, <a href="http://homedirs.ccs.neu.edu/types/resources/talks/preservation-types.pdf">slides</a>) explained why he is implementing a semantics for <a href="https://github.com/racket/typed-racket">Typed Racket</a> inspired by Michael Vitousek&rsquo;s work on <a href="http://homes.soic.indiana.edu/mvitouse/papers/popl17.pdf">Reticulated Python</a>. The &ldquo;why&rdquo; is &ldquo;performance&rdquo;. The Reticulated semantics will enforce a notion of tag soundness in kind of <a href="https://en.wikipedia.org/wiki/Deal_with_the_Devil">devils contract</a> to improve performance.</p></li> + <li> + <p><a href="https://cs.brown.edu/~ptunnell/">Preston Tunnell-Wilson</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tpk-crowdsource-lang-design/">ONWARD 2017</a>) recently sent questions about programming language design to <a href="https://www.mturk.com/mturk/welcome">Mechanical Turk</a> workers. Survey says, developers have extremely diverse opinions about what they <em>expect</em> and what they <em>want</em> regarding scope, inheritance, and infix operators.</p></li></ul> + +<p>In the early afternoon, we had two talks on similar themes as the morning session:</p> + +<ul> + <li> + <p><a href="https://github.com/akuhlens">Andre Kuhlenschmidt</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is exploring the design space of efficient implementations for run-time type checks. The main challenge is how to <em>monitor</em> higher-order data in a way that efficiently performs type checks and can help the programmer debug any failed checks. This talk presented data comparing two approaches to the program; I believe the latter, improved approach is based on <a href="http://homepages.inf.ed.ac.uk/wadler/papers/coercions/coercions.pdf">coercions</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) explained that there are many ways to adapt type soundness to a gradually typed language, and presented some data comparing Typed Racket&rsquo;s <em>generalized soudness</em> to Reticulated Python&rsquo;s <em>tag soundness</em>. The data suggests that tag soundness never adds an order-of-magnitude slowdown.</p></li></ul> + +<p>Next on the schedule were two talks about implementing advanced type systems in Racket&rsquo;s macro expander (think: meta-level linguistic re-use, capture-avoiding substitution for free!)</p> + +<ul> + <li> + <p><a href="https://github.com/iitalics">Milo Turner</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) first showed how to implement <a href="https://gankro.github.io/blah/linear-rust/#definitions-and-the-state-of-rust">linear and affine</a> type systems using <a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html">syntax-parse</a>, and second presented a simpler implementation using the <a href="http://docs.racket-lang.org/turnstile/index.html">Turnstile</a> library.</p></li> + <li> + <p><a href="http://www.davidchristiansen.dk/">David Christiansen</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is building <a href="https://github.com/david-christiansen/pudding">a proof assistant</a> in Racket. This talk focused on the design and implementation of proof tactics.</p></li></ul> + +<p>After a short break, we heard about something completely different:</p> + +<ul> + <li><a href="http://justinpombrio.net/">Justin Pombrio</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/research/plt/dl/icfp2017/">ICFP 2017</a>) taught us to interpet the scoping rules of a &ldquo;core&rdquo; language as a preorder. Using the preorder, he then showed how to <em>infer</em> the scoping rules of any &ldquo;surface&rdquo; language based on its translation to the &ldquo;core&rdquo;.</li></ul> + +<p>Last summer and fall, Jeremy Siek hosted two REUs (<a href="https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5517&amp;from=fund">research experience for undergraduates</a>) at Indiana University. The two students gave the next talks:</p> + +<ul> + <li> + <p>Di Zhong (<a href="https://www.cs.indiana.edu/">IU</a>) talked about implementing interpreters in Racket, Python, and Haskell. As I understand, this was a hands-on experience through <a href="https://www.cis.upenn.edu/~bcpierce/tapl/">TAPL</a> and <a href="https://redex.racket-lang.org/">the Redex book</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="https://www.cs.indiana.edu/">IU</a>) demonstrated her implementation of <a href="http://theory.stanford.edu/~aiken/publications/papers/popl94.pdf">conditional types</a> for <a href="https://github.com/mvitousek/reticulated">Reticulated</a>.</p></li></ul> + +<p>Finally,</p> + +<ul> + <li><a href="https://nikivazou.github.io/">Niki Vazou</a> (<a href="https://www.cs.umd.edu/">UMD</a>) presented a theory of gradual refinement types. Any &ldquo;holes&rdquo; in the refinements introduce a search problem; type checking attempts to solve the problem by finding a predicate that unifies a function definition and its callers.</li></ul> + +<p>This meeting was a great opportunity to reflect on the recent past and share opinions on what&rsquo;s worth pursuing in the future. Many thanks to the participants, and to the NSF for the support!</p> + +<blockquote> + <p>If you want to know about the future, you need to ask the young people who will create it. Young people don&rsquo;t know what can&rsquo;t be done, and so they go ahead and do it. &mdash; <a href="https://www.youtube.com/watch?v=sM1bNR4DmhU">Ivan Sutherland</a></p></blockquote> + + Categorical Semantics for Dynamically Typed Programming Languages + + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + 2017-05-01T12:25:17Z + 2017-05-01T12:25:17Z + + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> + + Gradual Typing Across the Spectrum + + urn:http-prl-ccs-neu-edu:-blog-2016-05-18-gradual-typing-across-the-spectrum + 2016-05-18T07:58:56Z + 2016-05-18T07:58:56Z + + Asumu Takikawa + +<blockquote> + <p>Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. &mdash; Matthias Felleisen</p></blockquote> + +<p>Yesterday we hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found <a href="http://prl.ccs.neu.edu/gtp/pi2016/pi2016.html">here</a>). In case you aren&rsquo;t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.</p> + +<p>Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing&mdash;the ideas that cross language and platform barriers.</p> +<!-- more--> + +<p>There were a good number of both institutions and programming languages represented at the meeting, with researchers from all of <a href="http://cs.brown.edu/research/plt/">Brown University</a>, <a href="https://wonks.github.io/">Indiana University</a>, <a href="http://prl.ccs.neu.edu/">Northeastern University</a>, and the <a href="http://www.cs.umd.edu/projects/PL/">University of Maryland</a>. The languages that we work on cover a broad subset of the dynamically-typed languages: Clojure, JavaScript, R, Racket, Ruby, Pyret, and Python.</p> + +<div class="figure"><img src="/img/2016-day-slide-4.png" alt="" /> + <p class="caption"></p></div> + +<p>The specific research artifacts that were represented include <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>, <a href="https://github.com/plum-umd/rdl">RDL</a> (contracts for Ruby), <a href="http://plg.uwaterloo.ca/~dynjs/strongscript/">StrongScript</a>, <a href="http://typedclojure.org/">Typed Clojure</a>, and <a href="http://docs.racket-lang.org/ts-guide/index.html">Typed Racket</a>.</p> + +<p>In this blog post, I&rsquo;ll summarize some of the key research themes that were brought up at the meeting. Since I can&rsquo;t go into too much detail about every topic, I will link to the relevant research papers and other resources.</p> + +<p>At a high level, the talks covered four major facets of gradual typing: expressiveness, performance, usability, and implementation techniques.</p> + +<h2 id="expressiveness">Expressiveness</h2> + +<p>By expressiveness, I mean what kinds of language features a gradual type system supports and the richness of the reasoning that the type system provides. Since gradual typing is about augmenting existing dynamically-typed languages, a gradual type system should support the language features that programmers actually use.</p> + +<p>This is why recent implementations of gradual typing have focused on enabling object-oriented programming, since objects are widely used in nearly all dynamically-typed languages in use today. Unfortunately, since different languages have wildly different object systems, it&rsquo;s hard to compare research on gradual OO languages. Ben Chung is working to address this by coming up with a formal model that tries to unify various accounts of objects in order to better explain the design tradeoffs. His goal is to cover the core ideas in Reticulated Python, StrongScript, and Typed Racket.</p> + +<p>Of course, dynamically-typed languages have a lot more than OO features. Along these lines, I gave a talk on how at NU we&rsquo;re working to extend Typed Racket to cover everything from objects (my thesis topic), first-class modules (Dan Feltey&rsquo;s MS project), and higher-order contracts (Brian LaChance&rsquo;s MS project).</p> + +<p>On the other side, as programs get more complex, programmers may wish to write richer type specifications that provide even more guarantees. This makes gradual typing a wide spectrum that goes from completely untyped, fully typed, and then beyond to dependently typed. Andrew Kent and David Christiansen both presented work that takes gradual typing beyond ordinary typed reasoning with dependent types.</p> + +<p>Andrew presented an extension of Typed Racket that adds type refinements that can check rich properties (like vector bounds) that are found in real Racket code (see his RacketCon <a href="https://www.youtube.com/watch?v=ejFJIAsvdEg">talk</a> and recent <a href="http://arxiv.org/pdf/1511.07033.pdf">PLDI paper</a>). David Christiansen followed with a talk about adding dependent type theory to Typed Racket, which would allow correct-by-construction programming using a Nuprl-like proof system (he had a very cool GUI proof assistant demo in his slides!).</p> + +<h2 id="performance">Performance</h2> + +<div class="figure"><img src="/img/2016-day-slide-8.png" alt="" /> + <p class="caption"></p></div> + +<p>One of the key practical concerns about gradual typing is its performance overhead. It&rsquo;s a concern because in order to ensure type safety, a gradually-typed language implementation needs to install dynamic checks between the typed and untyped parts of a program. This catches any inconsistencies between the typed interfaces and how the untyped code may call into them.</p> + +<p>Ben Greenman gave an upbeat talk that set the stage for this topic, pointing out some key lessons that we&rsquo;ve learned about performance from building Typed Racket. The main idea he presented (also the topic of our <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016 paper</a>) is that to evaluate a gradual type system, you want to explore different ways of adding types to a program and see how much it costs. This evaluation effort started with Typed Racket, but he and Zeina Migeed are working on expanding it to Reticulated Python.</p> + +<p>From IU, Andre Kuhlenschmidt and Deyaaeldeen Almahallawi are exploring how ahead-of-time (AOT) compilation strategies could help reduce the cost of gradual typing. In particular, they are working on implementing <a href="https://github.com/deyaaeldeen/Schml">Schml</a>: a compiler from the gradually-typed lambda calculus to C.</p> + +<p>In addition to AOT compilation, the folks at IU are exploring tracing JIT compilation as a means to make gradual typing faster. More specifically, Spenser Bauman talked about Pycket, an alternative implementation of Racket that uses RPython/PyPy to dramatically lower the overhead of gradual typing (also see the <a href="https://www.youtube.com/watch?v=GOfIY8NHAqg">recording</a> of Spenser&rsquo;s talk on the topic at RacketCon and his <a href="http://homes.soic.indiana.edu/samth/pycket-draft.pdf">ICFP paper</a>).</p> + +<h2 id="usability">Usability</h2> + +<p>On the usability side, both Shriram Krishnamurthi and Ambrose Bonnaire-Sergeant made observations on what it takes to get gradual typing in the hands of real software developers.</p> + +<p>Shriram approached the topic from the angle of CS education, which is the focus of the <a href="http://www.pyret.org">Pyret</a> language, and shared what the Brown language group is working on. While Pyret doesn&rsquo;t exactly fit the mold of gradual typing, it&rsquo;s a close cousin since it&rsquo;s a dynamically-typed language that explicitly takes design cues from the best parts of statically-typed languages. That approach lets CS beginners think in terms of types (the approach spearheaded by <a href="http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html">HtDP</a> and <a href="http://www.bootstrapworld.org/">Bootstrap</a>) without having to battle a typechecker from the start.</p> + +<p>For professional software developers, a major concern with gradual typing is that writing type annotations may be a tedious and time intensive task. Ambrose, who is the creator of Typed Clojure, shared some preliminary work on how to cut down on the tedium by inferring gradual type annotations by instrumenting programs for a dynamic analysis. The hope is to be able to infer both recursive and polymorphic type annotations automatically from tests (you may also be interested in Ambrose&rsquo;s recent <a href="http://frenchy64.github.io/papers/esop16-short.pdf">ESOP paper</a> on Typed Clojure).</p> + +<h2 id="implementation-techniques">Implementation Techniques</h2> + +<p>Finally, several talks focused on alternative implementation techniques for gradual typing that provide a variety of software engineering benefits for implementers.</p> + +<p>From Maryland, Brianna Ren gave a talk on Hummingbird, a just-in-time typechecker for Ruby programs (also see the upcoming <a href="http://www.cs.umd.edu/~jfoster/papers/pldi16.pdf">PLDI paper</a> by Brianna and Jeff Foster). The basic idea is that it&rsquo;s hard to implement a traditional static typechecker for a language that heavily relies on metaprogramming, in which the fields/methods of classes may be rewritten at run-time. This is particularly tricky for frameworks like Ruby on Rails. Instead of checking types at compile-time, Hummingbird actually executes the typechecker at run-time in order to be able to accurately check programs that use run-time metaprogramming. To reduce overheads, she uses a cache for typechecking that is invalidated when classes are modified.</p> + +<p>Stephen Chang gave a very different view on metaprogramming in his talk, which focused on <em>implementing</em> typecheckers using metaprogramming (the <a href="http://docs.racket-lang.org/trivial/index.html">trivial</a> Typed Racket library is an offshoot of this work). His key idea is that typecheckers share many aspects with macro-based metaprogramming systems, such as the need to traverse syntax and annotate it with information. Since they share so much in common, why not just implement the typechecker as a macro? Stephen demonstrates that not only is this possible, but that it&rsquo;s possible to implement a wide variety of type system features this way including (local) type inference. The connection to gradual typing is that even a gradual type system can be implemented as a metaprogram by integrating the generation of dynamic checks into the macro transformation process.</p> + +<p>The last talk of the day (but certainly not the least), was by Michael Vitousek, who focused on the <em>transient</em> implementation of gradual typing (first described in his <a href="http://homes.soic.indiana.edu/mvitouse/papers/dls14.pdf">DLS paper</a>). Traditionally, gradual type systems have implemented their dynamic checks using <a href="https://en.wikipedia.org/wiki/Proxy_pattern">proxy</a> objects that wrap method implementations with both pre- and post-checks. Unfortunately, this implementation technique often conflicts with the underlying language. Since proxying changes the identity of an object it can interfere with object equality tests. Instead, the transient approach bakes the dynamic checks into and throughout the typed code to implement a &ldquo;defense in depth&rdquo; against inconsistencies with untyped code. The great thing about this implementation technique is that it doesn&rsquo;t demand any specialized support from the underlying language runtime and is therefore easy to port to other languages (like JavaScript).</p> + +<h2 id="conclusion">Conclusion</h2> + +<div class="figure"><img src="/img/2016-day-slide-3.png" alt="" /> + <p class="caption"></p></div> + +<p>Hopefully this blog post helps provide a better picture of the state of gradual typing research. The exciting thing about gradual typing is that it contains both interesting theoretical problems and also connects to the practical needs of software developers.</p> \ No newline at end of file diff --git a/blog/feeds/gradual-typing.rss.xml b/blog/feeds/gradual-typing.rss.xml new file mode 100644 index 00000000..c3d8a3e0 --- /dev/null +++ b/blog/feeds/gradual-typing.rss.xml @@ -0,0 +1,521 @@ + + + + PRL Blog: Posts tagged 'gradual typing' + PRL Blog: Posts tagged 'gradual typing' + http://prl.ccs.neu.edu/blog/tags/gradual-typing.html + Thu, 31 Oct 2019 21:58:26 UT + Thu, 31 Oct 2019 21:58:26 UT + 1800 + + Complete Monitors for Gradual Types + http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/?utm_source=gradual-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + Thu, 31 Oct 2019 21:58:26 UT + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + The Behavior of Gradual Types: A User Study + http://prl.ccs.neu.edu/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/?utm_source=gradual-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + Tue, 11 Dec 2018 19:50:33 UT + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + A Spectrum of Type Soundness and Performance + http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/?utm_source=gradual-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-06-a-spectrum-of-type-soundness-and-performance + Sat, 06 Oct 2018 11:23:35 UT + Ben Greenman + +<p>The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>A Spectrum of Type Soundness and Performance</em> by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018">http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018</a></p></blockquote> + +<p>A dynamically-typed language runs any program that &ldquo;looks good&rdquo; (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both &ldquo;looks good&rdquo; and is well-typed according to a type checker.</p> + +<p>A <em>mixed-typed</em> language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.</p> + +<div class="figure"><img src="/img/mixed-typed-systems-by-year.png" alt="Figure 1: Some mixed-typed languages" /> + <p class="caption">Figure 1: Some mixed-typed languages</p></div> + +<p>These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:</p> + +<ul> + <li><strong>MACLISP</strong> defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the <a href="http://www.softwarepreservation.org/projects/LISP/MIT/Moon-MACLISP_Reference_Manual-Apr_08_1974.pdf">Moonual</a>. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).</li> + <li><strong>Strongtalk</strong> includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a> position paper.</li> + <li><strong>Typed Racket</strong> lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.</li> + <li><strong>Thorn</strong> requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.</li> + <li><strong>Reticulated</strong> lets a program combine static and dynamic <em>expressions</em> and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek <a href="https://dl.acm.org/citation.cfm?id=3009849">POPL 2017</a>).</li></ul> + +<p>That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system&rsquo;s protocol for checking dynamically-typed values at a <em>type boundary</em> (between static and dynamic code).</p> + +<h3 id="main-contribution">Main Contribution</h3> + +<p>In the paper <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>A Spectrum of Type Soundness and Performance</em></a>, we define a tiny mixed-typed language and show three ways to define the behavior of programs &mdash; based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.</p> + +<p>The three behaviors are inspired by existing languages. A <strong>higher order</strong> behavior ensures that dynamically-typed values match the static type at a boundary &mdash; by checking the value when possible, and by monitoring the value&rsquo;s future interactions when necessary. A <strong>first order</strong> behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An <strong>erasure</strong> behavior does no checking whatsoever.</p> + +<blockquote> + <p>Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function <code>f</code>, then one way to enforce the type boundary is to wrap <code>f</code> in a proxy to assert that every future call to <code>f</code> returns a number. In this case, the proxy monitors the behavior of <code>f</code>.</p></blockquote> + +<p>Concretely, the paper defines three formal semantics with the same names. The <strong>higher-order</strong> semantics enforces full types at the boundaries (Section 2.3). The <strong>first-order</strong> semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every &ldquo;selector&rdquo; operation in typed code, e.g., function application, data structure access (Section 2.5). The <strong>erasure</strong> semantics simply ignores the types (Section 2.4).</p> + +<p>Each semantics satisfies a <em>different</em> notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">online supplement</a> gives full proofs.</p> + +<p>The paper has more to say about the meta-theory. See section 2 and section 4.</p> + +<blockquote> + <p>To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for <a href="https://dl.acm.org/citation.cfm?id=2676971">subset of the language</a> (in the spirit of <a href="http://soundiness.org/">soundiness</a>) or use the name &ldquo;type soundness&rdquo; to describe <a href="https://dl.acm.org/citation.cfm?id=2676971">a different property</a>.</p></blockquote> + +<p>Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations &mdash; similar to Typed Racket&rsquo;s <a href="http://docs.racket-lang.org/ts-reference/Typed_Racket_Syntax_Without_Type_Checking.html">no-check</a> language.</p> + +<p>Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (<a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016</a>). The programs we measured are the non-object-oriented ones from our <a href="http://docs.racket-lang.org/gtp-benchmarks/index.html">benchmark suite</a>.</p> + +<p>To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises &mdash; see section 3 of the paper, section B of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, and/or the <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slides</a>.</p> + +<h3 id="implications">Implications</h3> + +<ol> + <li>The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, or <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slide 114</a>.</li> + <li>Programmers using mixed-typed languages need to know what guarantees their types provide. (It is <a href="https://twitter.com/jbandi/status/965005464638541825">not safe to assume that TypeScript types give the same guarantees as OCaml types</a>!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.</li> + <li>The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?</li></ol> + +<blockquote> + <p>Note: in this post, a <em>mixed-typed language</em> is one that allows any combination of static and dynamic typing. A <em>gradually-typed language</em> is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>).</p></blockquote> + + Sampling Gradual Typing Performance + http://prl.ccs.neu.edu/blog/2018/05/08/sampling-gradual-typing-performance/?utm_source=gradual-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + Tue, 08 May 2018 15:37:37 UT + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> + + Gradual Typing Across the Spectrum, part II + http://prl.ccs.neu.edu/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/?utm_source=gradual-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-22-gradual-typing-across-the-spectrum-part-ii + Tue, 22 Aug 2017 15:54:06 UT + Ben Greenman + +<p>Last week, Northeastern hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:</p> + +<p><a href="http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html">http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html</a></p> + +<p>A common thread among the talks was the question: <em>how to convert a research idea into a tool for software developers?</em></p> +<!-- more--> + +<p>In my mind, gradual typing <em>is</em> an answer to one instance of this question. The research idea is strong static type systems, and the software developers are the millions using dynamically typed languages. I know that static typing can make programs easier to write and maintain. The developers know that dynamic typing has benefits; moreover they know better than to migrate their code from one language to another on a whim. Gradual typing is a linguistic solution to the problem of <em>adding</em> the benefits of static typing to a dynamically typed language.</p> + +<p>Enough opinions, let&rsquo;s talk about the talks.</p> + +<p>The morning session consisted of four talks:</p> + +<ul> + <li> + <p><a href="https://www.cs.umd.edu/people/milod">Milod Kazerounian</a> (<a href="https://www.cs.umd.edu/">UMD</a>) spoke about upgrading the <a href="https://github.com/plum-umd/rdl">RDL</a> type checker for Ruby with support for refinement types. The idea is to compile Ruby code and types to <a href="https://emina.github.io/rosette/">Rosette</a>, and profit from <a href="http://yices.csl.sri.com/papers/cav2007.pdf">SMT</a>-assisted type checking.</p></li> + <li> + <p><a href="http://ambrosebs.com/">Ambrose Bonnaire-Sergeant</a> (<a href="https://www.cs.indiana.edu/">IU</a>, <a href="http://ambrosebs.com/talks/squash-work-boston-pi-2017.pdf">slides</a>) has been inferring <em>useful</em> <a href="http://typedclojure.org/">Typed Clojure</a> types through dynamic analysis of Clojure programs. His tool observes how values flow through a program at run-time, then lifts these observations into possibly-recursive, possibly-incorrect type annotations. The surprising result is that the tool quickly (1&ndash;2 seconds per unit test, I think) infers types that can help a developer start annotating a program.</p></li> + <li> + <p><a href="http://ccs.neu.edu/~types/">Ben Greenman</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>, <a href="http://homedirs.ccs.neu.edu/types/resources/talks/preservation-types.pdf">slides</a>) explained why he is implementing a semantics for <a href="https://github.com/racket/typed-racket">Typed Racket</a> inspired by Michael Vitousek&rsquo;s work on <a href="http://homes.soic.indiana.edu/mvitouse/papers/popl17.pdf">Reticulated Python</a>. The &ldquo;why&rdquo; is &ldquo;performance&rdquo;. The Reticulated semantics will enforce a notion of tag soundness in kind of <a href="https://en.wikipedia.org/wiki/Deal_with_the_Devil">devils contract</a> to improve performance.</p></li> + <li> + <p><a href="https://cs.brown.edu/~ptunnell/">Preston Tunnell-Wilson</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tpk-crowdsource-lang-design/">ONWARD 2017</a>) recently sent questions about programming language design to <a href="https://www.mturk.com/mturk/welcome">Mechanical Turk</a> workers. Survey says, developers have extremely diverse opinions about what they <em>expect</em> and what they <em>want</em> regarding scope, inheritance, and infix operators.</p></li></ul> + +<p>In the early afternoon, we had two talks on similar themes as the morning session:</p> + +<ul> + <li> + <p><a href="https://github.com/akuhlens">Andre Kuhlenschmidt</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is exploring the design space of efficient implementations for run-time type checks. The main challenge is how to <em>monitor</em> higher-order data in a way that efficiently performs type checks and can help the programmer debug any failed checks. This talk presented data comparing two approaches to the program; I believe the latter, improved approach is based on <a href="http://homepages.inf.ed.ac.uk/wadler/papers/coercions/coercions.pdf">coercions</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) explained that there are many ways to adapt type soundness to a gradually typed language, and presented some data comparing Typed Racket&rsquo;s <em>generalized soudness</em> to Reticulated Python&rsquo;s <em>tag soundness</em>. The data suggests that tag soundness never adds an order-of-magnitude slowdown.</p></li></ul> + +<p>Next on the schedule were two talks about implementing advanced type systems in Racket&rsquo;s macro expander (think: meta-level linguistic re-use, capture-avoiding substitution for free!)</p> + +<ul> + <li> + <p><a href="https://github.com/iitalics">Milo Turner</a> (<a href="http://www.ccis.northeastern.edu/">NEU</a>) first showed how to implement <a href="https://gankro.github.io/blah/linear-rust/#definitions-and-the-state-of-rust">linear and affine</a> type systems using <a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html">syntax-parse</a>, and second presented a simpler implementation using the <a href="http://docs.racket-lang.org/turnstile/index.html">Turnstile</a> library.</p></li> + <li> + <p><a href="http://www.davidchristiansen.dk/">David Christiansen</a> (<a href="https://www.cs.indiana.edu/">IU</a>) is building <a href="https://github.com/david-christiansen/pudding">a proof assistant</a> in Racket. This talk focused on the design and implementation of proof tactics.</p></li></ul> + +<p>After a short break, we heard about something completely different:</p> + +<ul> + <li><a href="http://justinpombrio.net/">Justin Pombrio</a> (<a href="http://cs.brown.edu/">Brown</a>, <a href="http://cs.brown.edu/research/plt/dl/icfp2017/">ICFP 2017</a>) taught us to interpet the scoping rules of a &ldquo;core&rdquo; language as a preorder. Using the preorder, he then showed how to <em>infer</em> the scoping rules of any &ldquo;surface&rdquo; language based on its translation to the &ldquo;core&rdquo;.</li></ul> + +<p>Last summer and fall, Jeremy Siek hosted two REUs (<a href="https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5517&amp;from=fund">research experience for undergraduates</a>) at Indiana University. The two students gave the next talks:</p> + +<ul> + <li> + <p>Di Zhong (<a href="https://www.cs.indiana.edu/">IU</a>) talked about implementing interpreters in Racket, Python, and Haskell. As I understand, this was a hands-on experience through <a href="https://www.cis.upenn.edu/~bcpierce/tapl/">TAPL</a> and <a href="https://redex.racket-lang.org/">the Redex book</a>.</p></li> + <li> + <p><a href="https://zeinamigeed.com/">Zeina Migeed</a> (<a href="https://www.cs.indiana.edu/">IU</a>) demonstrated her implementation of <a href="http://theory.stanford.edu/~aiken/publications/papers/popl94.pdf">conditional types</a> for <a href="https://github.com/mvitousek/reticulated">Reticulated</a>.</p></li></ul> + +<p>Finally,</p> + +<ul> + <li><a href="https://nikivazou.github.io/">Niki Vazou</a> (<a href="https://www.cs.umd.edu/">UMD</a>) presented a theory of gradual refinement types. Any &ldquo;holes&rdquo; in the refinements introduce a search problem; type checking attempts to solve the problem by finding a predicate that unifies a function definition and its callers.</li></ul> + +<p>This meeting was a great opportunity to reflect on the recent past and share opinions on what&rsquo;s worth pursuing in the future. Many thanks to the participants, and to the NSF for the support!</p> + +<blockquote> + <p>If you want to know about the future, you need to ask the young people who will create it. Young people don&rsquo;t know what can&rsquo;t be done, and so they go ahead and do it. &mdash; <a href="https://www.youtube.com/watch?v=sM1bNR4DmhU">Ivan Sutherland</a></p></blockquote> + + Categorical Semantics for Dynamically Typed Programming Languages + http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/?utm_source=gradual-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-01-categorical-semantics-for-dynamically-typed-programming-languages + Mon, 01 May 2017 12:25:17 UT + Max New + <!-- more--> + +<p>In 1969, Dana Scott wrote an <a href="/blog/static/scott-69-93-type-theoretical-alternative.pdf">unpublished manuscript</a> in which he said untyped lambda calculus had no mathematical meaning, 11 years later he wrote <a href="/blog/static/scott-80-relating-theories.pdf">a paper</a> that organized many of the different semantics he and others had since found using the language of category theory.</p> + +<p>This latter paper is really the first deserving of the title &ldquo;categorical semantics of dynamic typing&rdquo;, and so I&rsquo;m going to present some of the theorems and &ldquo;theorems&rdquo; presented in that paper, but mingled with the history of the idea and the preceding papers that led to them.</p> + +<p><a href="/blog/static/dyn-cats.pdf">My Full Notes</a> continue the story, and you might also be interested in the <a href="https://github.com/nuprl/hopl-s2017/blob/master/lecture_notes/2017-04-07.md">discussion during the lecture</a>.</p> + + Gradual Typing Across the Spectrum + http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/?utm_source=gradual-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-18-gradual-typing-across-the-spectrum + Wed, 18 May 2016 07:58:56 UT + Asumu Takikawa + +<blockquote> + <p>Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. &mdash; Matthias Felleisen</p></blockquote> + +<p>Yesterday we hosted a PI meeting for the <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found <a href="http://prl.ccs.neu.edu/gtp/pi2016/pi2016.html">here</a>). In case you aren&rsquo;t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.</p> + +<p>Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing&mdash;the ideas that cross language and platform barriers.</p> +<!-- more--> + +<p>There were a good number of both institutions and programming languages represented at the meeting, with researchers from all of <a href="http://cs.brown.edu/research/plt/">Brown University</a>, <a href="https://wonks.github.io/">Indiana University</a>, <a href="http://prl.ccs.neu.edu/">Northeastern University</a>, and the <a href="http://www.cs.umd.edu/projects/PL/">University of Maryland</a>. The languages that we work on cover a broad subset of the dynamically-typed languages: Clojure, JavaScript, R, Racket, Ruby, Pyret, and Python.</p> + +<div class="figure"><img src="/img/2016-day-slide-4.png" alt="" /> + <p class="caption"></p></div> + +<p>The specific research artifacts that were represented include <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>, <a href="https://github.com/plum-umd/rdl">RDL</a> (contracts for Ruby), <a href="http://plg.uwaterloo.ca/~dynjs/strongscript/">StrongScript</a>, <a href="http://typedclojure.org/">Typed Clojure</a>, and <a href="http://docs.racket-lang.org/ts-guide/index.html">Typed Racket</a>.</p> + +<p>In this blog post, I&rsquo;ll summarize some of the key research themes that were brought up at the meeting. Since I can&rsquo;t go into too much detail about every topic, I will link to the relevant research papers and other resources.</p> + +<p>At a high level, the talks covered four major facets of gradual typing: expressiveness, performance, usability, and implementation techniques.</p> + +<h2 id="expressiveness">Expressiveness</h2> + +<p>By expressiveness, I mean what kinds of language features a gradual type system supports and the richness of the reasoning that the type system provides. Since gradual typing is about augmenting existing dynamically-typed languages, a gradual type system should support the language features that programmers actually use.</p> + +<p>This is why recent implementations of gradual typing have focused on enabling object-oriented programming, since objects are widely used in nearly all dynamically-typed languages in use today. Unfortunately, since different languages have wildly different object systems, it&rsquo;s hard to compare research on gradual OO languages. Ben Chung is working to address this by coming up with a formal model that tries to unify various accounts of objects in order to better explain the design tradeoffs. His goal is to cover the core ideas in Reticulated Python, StrongScript, and Typed Racket.</p> + +<p>Of course, dynamically-typed languages have a lot more than OO features. Along these lines, I gave a talk on how at NU we&rsquo;re working to extend Typed Racket to cover everything from objects (my thesis topic), first-class modules (Dan Feltey&rsquo;s MS project), and higher-order contracts (Brian LaChance&rsquo;s MS project).</p> + +<p>On the other side, as programs get more complex, programmers may wish to write richer type specifications that provide even more guarantees. This makes gradual typing a wide spectrum that goes from completely untyped, fully typed, and then beyond to dependently typed. Andrew Kent and David Christiansen both presented work that takes gradual typing beyond ordinary typed reasoning with dependent types.</p> + +<p>Andrew presented an extension of Typed Racket that adds type refinements that can check rich properties (like vector bounds) that are found in real Racket code (see his RacketCon <a href="https://www.youtube.com/watch?v=ejFJIAsvdEg">talk</a> and recent <a href="http://arxiv.org/pdf/1511.07033.pdf">PLDI paper</a>). David Christiansen followed with a talk about adding dependent type theory to Typed Racket, which would allow correct-by-construction programming using a Nuprl-like proof system (he had a very cool GUI proof assistant demo in his slides!).</p> + +<h2 id="performance">Performance</h2> + +<div class="figure"><img src="/img/2016-day-slide-8.png" alt="" /> + <p class="caption"></p></div> + +<p>One of the key practical concerns about gradual typing is its performance overhead. It&rsquo;s a concern because in order to ensure type safety, a gradually-typed language implementation needs to install dynamic checks between the typed and untyped parts of a program. This catches any inconsistencies between the typed interfaces and how the untyped code may call into them.</p> + +<p>Ben Greenman gave an upbeat talk that set the stage for this topic, pointing out some key lessons that we&rsquo;ve learned about performance from building Typed Racket. The main idea he presented (also the topic of our <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016 paper</a>) is that to evaluate a gradual type system, you want to explore different ways of adding types to a program and see how much it costs. This evaluation effort started with Typed Racket, but he and Zeina Migeed are working on expanding it to Reticulated Python.</p> + +<p>From IU, Andre Kuhlenschmidt and Deyaaeldeen Almahallawi are exploring how ahead-of-time (AOT) compilation strategies could help reduce the cost of gradual typing. In particular, they are working on implementing <a href="https://github.com/deyaaeldeen/Schml">Schml</a>: a compiler from the gradually-typed lambda calculus to C.</p> + +<p>In addition to AOT compilation, the folks at IU are exploring tracing JIT compilation as a means to make gradual typing faster. More specifically, Spenser Bauman talked about Pycket, an alternative implementation of Racket that uses RPython/PyPy to dramatically lower the overhead of gradual typing (also see the <a href="https://www.youtube.com/watch?v=GOfIY8NHAqg">recording</a> of Spenser&rsquo;s talk on the topic at RacketCon and his <a href="http://homes.soic.indiana.edu/samth/pycket-draft.pdf">ICFP paper</a>).</p> + +<h2 id="usability">Usability</h2> + +<p>On the usability side, both Shriram Krishnamurthi and Ambrose Bonnaire-Sergeant made observations on what it takes to get gradual typing in the hands of real software developers.</p> + +<p>Shriram approached the topic from the angle of CS education, which is the focus of the <a href="http://www.pyret.org">Pyret</a> language, and shared what the Brown language group is working on. While Pyret doesn&rsquo;t exactly fit the mold of gradual typing, it&rsquo;s a close cousin since it&rsquo;s a dynamically-typed language that explicitly takes design cues from the best parts of statically-typed languages. That approach lets CS beginners think in terms of types (the approach spearheaded by <a href="http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html">HtDP</a> and <a href="http://www.bootstrapworld.org/">Bootstrap</a>) without having to battle a typechecker from the start.</p> + +<p>For professional software developers, a major concern with gradual typing is that writing type annotations may be a tedious and time intensive task. Ambrose, who is the creator of Typed Clojure, shared some preliminary work on how to cut down on the tedium by inferring gradual type annotations by instrumenting programs for a dynamic analysis. The hope is to be able to infer both recursive and polymorphic type annotations automatically from tests (you may also be interested in Ambrose&rsquo;s recent <a href="http://frenchy64.github.io/papers/esop16-short.pdf">ESOP paper</a> on Typed Clojure).</p> + +<h2 id="implementation-techniques">Implementation Techniques</h2> + +<p>Finally, several talks focused on alternative implementation techniques for gradual typing that provide a variety of software engineering benefits for implementers.</p> + +<p>From Maryland, Brianna Ren gave a talk on Hummingbird, a just-in-time typechecker for Ruby programs (also see the upcoming <a href="http://www.cs.umd.edu/~jfoster/papers/pldi16.pdf">PLDI paper</a> by Brianna and Jeff Foster). The basic idea is that it&rsquo;s hard to implement a traditional static typechecker for a language that heavily relies on metaprogramming, in which the fields/methods of classes may be rewritten at run-time. This is particularly tricky for frameworks like Ruby on Rails. Instead of checking types at compile-time, Hummingbird actually executes the typechecker at run-time in order to be able to accurately check programs that use run-time metaprogramming. To reduce overheads, she uses a cache for typechecking that is invalidated when classes are modified.</p> + +<p>Stephen Chang gave a very different view on metaprogramming in his talk, which focused on <em>implementing</em> typecheckers using metaprogramming (the <a href="http://docs.racket-lang.org/trivial/index.html">trivial</a> Typed Racket library is an offshoot of this work). His key idea is that typecheckers share many aspects with macro-based metaprogramming systems, such as the need to traverse syntax and annotate it with information. Since they share so much in common, why not just implement the typechecker as a macro? Stephen demonstrates that not only is this possible, but that it&rsquo;s possible to implement a wide variety of type system features this way including (local) type inference. The connection to gradual typing is that even a gradual type system can be implemented as a metaprogram by integrating the generation of dynamic checks into the macro transformation process.</p> + +<p>The last talk of the day (but certainly not the least), was by Michael Vitousek, who focused on the <em>transient</em> implementation of gradual typing (first described in his <a href="http://homes.soic.indiana.edu/mvitouse/papers/dls14.pdf">DLS paper</a>). Traditionally, gradual type systems have implemented their dynamic checks using <a href="https://en.wikipedia.org/wiki/Proxy_pattern">proxy</a> objects that wrap method implementations with both pre- and post-checks. Unfortunately, this implementation technique often conflicts with the underlying language. Since proxying changes the identity of an object it can interfere with object equality tests. Instead, the transient approach bakes the dynamic checks into and throughout the typed code to implement a &ldquo;defense in depth&rdquo; against inconsistencies with untyped code. The great thing about this implementation technique is that it doesn&rsquo;t demand any specialized support from the underlying language runtime and is therefore easy to port to other languages (like JavaScript).</p> + +<h2 id="conclusion">Conclusion</h2> + +<div class="figure"><img src="/img/2016-day-slide-3.png" alt="" /> + <p class="caption"></p></div> + +<p>Hopefully this blog post helps provide a better picture of the state of gradual typing research. The exciting thing about gradual typing is that it contains both interesting theoretical problems and also connects to the practical needs of software developers.</p> \ No newline at end of file diff --git a/blog/feeds/haskell.atom.xml b/blog/feeds/haskell.atom.xml new file mode 100644 index 00000000..6f67602e --- /dev/null +++ b/blog/feeds/haskell.atom.xml @@ -0,0 +1,456 @@ + + + PRL Blog: Posts tagged 'haskell' + + + urn:http-prl-ccs-neu-edu:-blog-tags-haskell-html + 2016-05-24T10:51:34Z + + Measuring GC latencies in Haskell, OCaml, Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + 2016-05-24T10:51:34Z + 2016-05-24T10:51:34Z + + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/haskell.rss.xml b/blog/feeds/haskell.rss.xml new file mode 100644 index 00000000..e4c5263c --- /dev/null +++ b/blog/feeds/haskell.rss.xml @@ -0,0 +1,456 @@ + + + + PRL Blog: Posts tagged 'haskell' + PRL Blog: Posts tagged 'haskell' + http://prl.ccs.neu.edu/blog/tags/haskell.html + Tue, 24 May 2016 10:51:34 UT + Tue, 24 May 2016 10:51:34 UT + 1800 + + Measuring GC latencies in Haskell, OCaml, Racket + http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/?utm_source=haskell&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + Tue, 24 May 2016 10:51:34 UT + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/higher-order-contracts.atom.xml b/blog/feeds/higher-order-contracts.atom.xml new file mode 100644 index 00000000..50a3cfdc --- /dev/null +++ b/blog/feeds/higher-order-contracts.atom.xml @@ -0,0 +1,186 @@ + + + PRL Blog: Posts tagged 'higher-order contracts' + + + urn:http-prl-ccs-neu-edu:-blog-tags-higher-order-contracts-html + 2019-04-07T23:15:11Z + + Forgetful and Heedful contracts + + urn:http-prl-ccs-neu-edu:-blog-2019-04-07-forgetful-and-heedful-contracts + 2019-04-07T23:15:11Z + 2019-04-07T23:15:11Z + + Ben Greenman + +<p><em>Forgetful</em> and <em>heedful</em> are two methods for space-efficient contracts developed by <a href="http://www.cs.pomona.edu/~michael/">Michael Greenberg</a> in <a href="https://arxiv.org/abs/1410.2813">2014</a>. These methods were born in the shadow of a third method, <em>eidetic</em>, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.</p> +<!-- more--> + +<hr /> + +<p>Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of <a href="https://www2.ccs.neu.edu/racket/pubs/popl11-dfff.pdf">three parties</a>: the API code, the client code, and the contract between them.</p> + +<p>For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:</p> + +<pre><code> #lang racket/base + + (module sort racket + (provide + (contract-out + [quicksort + (-&gt; (vectorof point/c) void?)])) + + (define point/c (vectorof integer?)) + + (define (quicksort points) + ....)) + + (module client racket + (require (submod ".." sort)) + (quicksort '())) + + (require 'client)</code></pre> + +<pre><code>quicksort: contract violation; + expected a vector + given: '() + in: the 1st argument of + (-&gt; (vectorof (vectorof integer?)) void?) + contract from: + (file.rkt sort) + blaming: (file.rkt client) + (assuming the contract is correct)</code></pre> + +<p>That covers the basics. For an extended introduction to contracts, visit <a href="https://docs.racket-lang.org/guide/contracts.html">The Racket Guide</a>.</p> + +<p>The quicksort example and the related figures are from the paper <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts: Fixing a Pathology of Gradual Typing</em></a></p> + +<h3 id="classic-contracts-and-space-efficiency">Classic contracts and &ldquo;Space Efficiency&rdquo;</h3> + +<p>The <code>(vectorof point/c)</code> contract used above describes a possibly-mutable array whose elements match the <code>point/c</code> contract. Since the array can be mutated, this contract has implications for two parties:</p> + +<ol> + <li>the client module must supply a good array, and</li> + <li>the sorting module must not insert a bad element.</li></ol> + +<p>To enforce the second condition, the <code>vectorof</code> contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:</p> + +<pre><code>(quicksort (vector (vector 4 4) + (vector 2 2) + (vector 1 1) + (vector 3 3)))</code></pre> + +<p>After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a <strong>solid</strong> blue bar for the <strong>write</strong> checks against the sort module and a <em>striped</em> blue bar for the <em>read</em> checks against the client.</p> + +<p><img src="/img/vector-chaperone-0.png" alt="A wrapped vector" /></p> + +<p>In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted &mdash; because every <strong>write</strong> and <em>read</em> applies a contract to the element.</p> + +<p><img src="/img/vector-chaperone-1.png" alt="Layers of element wrappers" /></p> + +<p>On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.</p> + +<p>Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and &ldquo;indirecting&rdquo; future writes/reads through wrappers. These space and time costs can add up.</p> + +<blockquote> + <p>&ldquo;on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times&rdquo; &mdash; <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts</em></a></p></blockquote> + +<p>To fix the problem, researchers have been exploring <em>space-efficient</em> implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and <em>eidetic</em>, <em>forgetful</em>, and <em>heedful</em> are his names for three implementations.</p> + +<p>(Although the goal of this post is to promote <em>forgetful</em> and <em>heedful</em>, we will review all three.)</p> + +<h3 id="eidetic-space-efficiency">Eidetic space-efficiency</h3> + +<p>The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a <em>merge</em> operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal &ldquo;wrapping&rdquo; contracts and their size is bounded by the number (and height) of source-code contracts in the program.</p> + +<p>An eidetic contract is an <code>N</code>-ary tree (for <code>N &gt; 0</code>):</p> + +<ul> + <li>each node represents a higher-order contract combinator, such as <code>vectorof</code></li> + <li>the <code>N</code> children of a node represent the different interactions that the value supports</li> + <li>each leaf is a list of non-higher-order, or <em>flat</em>, contracts</li></ul> + +<p>For example, the <code>(vectorof point/c)</code> source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf">Collapsible Contracts</a> paper has an illustration. Each tree node represents a <code>vectorof</code> contract; these nodes have <code>N=2</code> children because vectors support reads and writes.</p> + +<p>A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists &mdash; if an implementation comes with a <code>contract-stronger?</code> predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.</p> + +<p>Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should <code>contract-stronger?</code> try to solve problems that lack polynomial-time solutions?</p> + +<p>Thankfully, there are at least two &ldquo;compromise&rdquo; alternatives.</p> + +<h3 id="forgetful-space-efficiency">Forgetful space-efficiency</h3> +<!-- "no operation relies on e being a T2, skipping the check doesn't risk soundness" p.12--> +<!-- "In forgetful \lambda_H, we offer a simple solution to space inefficient casts: just forget about them" p.11--> +<!-- "Just the same, when accumulating casts on the stack, we throw away all but the last cast" p.11--> +<!-- "forgetful ... skip[s] checks and change[s] blame labels" p.3--> + +<blockquote> + <p>&ldquo;Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 10, bottom of page 23--></blockquote> + +<p>The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:</p> + +<ol> + <li>the client obligations from the old contract, and</li> + <li>the server obligations from the new contract</li></ol> + +<p>If not, proceed as usual &mdash; by wrapping (an unwrapped value) or raising an error. Every value receives at most <strong>one</strong> wrapper; this wrapper changes as the value flows to different clients.</p> + +<p>Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module <code>A</code> exports a function <code>f</code> with contract <code>(-&gt; T1 T2)</code> to module <code>B</code>, and suppose module <code>B</code> shares this function with a few other client modules using different contracts. As <code>f</code> flows to a new client, it keeps the <code>T1</code> domain check and gets a replacement for the <code>T2</code> codomain check.</p> + +<ul> + <li>Keeping <code>T1</code> ensures that the code inside the function (defined by module <code>A</code>) receives input that matches its expectation.</li> + <li>Replacing <code>T2</code> ensures that each new client receives output that it expects.</li></ul> + +<p>Unfortunately, replacing <code>T2</code> also means that clients of module <code>B</code> cannot trust the <code>T2</code> contract. This contract is not checked, and so forgetful contracts <strong>miss</strong> some errors that would be caught by standard contracts. For the same reason, a bug in module <code>B</code> may go undetected by its clients &mdash; even if a later contract reports an issue, the contract system has no memory that <code>B</code> was partly-responsible.</p> + +<p>Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.</p> + +<h3 id="heedful-space-efficiency">Heedful space-efficiency</h3> + +<p>A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set &mdash; or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.</p> + +<p>To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.</p> + +<p>The heedful method is a compromise between forgetful and eidetic.</p> + +<ul> + <li> + <p>Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of <code>contract-stronger?</code> predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.</p></li> + <li> + <p>Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.</p></li></ul> + +<p>For details, see <a href="https://arxiv.org/abs/1410.2813">the extended version</a> of Michael&rsquo;s POPL 2015 paper. Don&rsquo;t bother searching <a href="http://www.cs.pomona.edu/~michael/papers/popl2015_space.pdf">the conference version</a> &mdash; aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.</p> + +<p><code>*</code> If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to <a href="http://mballantyne.net/">Michael Ballantyne</a> for explaining this to me.</p> + +<h3 id="priorities-and-appearances">Priorities and Appearances</h3> + +<p>The extended version of <em>Space-Efficient Manifest Contracts</em> introduces the forgetful and heedful methods with extreme modesty. It&rsquo;s tempting to skip past them and focus on the eidetic method.</p> + +<blockquote> + <p>&ldquo;Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [<a href="https://dl.acm.org/citation.cfm?doid=1706299.1706342">Siek and Wadler 2010</a>]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen&mdash;warm up exercises for eidetic.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 1, bottom of page 2--></blockquote> + +<p>And yet, at least two other research papers rely on these &ldquo;strawmen&rdquo; &mdash; or rather, the ideas behind the names.</p> + +<p><a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a>, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.</p> + +<blockquote> + <p>&ldquo;if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- page 21--></blockquote> + +<p>This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.</p> + +<blockquote> + <p>&ldquo;removing these casts preserves the soundness of the evaluation while reducing the number of them&rdquo;</p> + <p>&ldquo;while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- p.27--><!-- page 21--></blockquote> +<!-- The followup at POPL 2019 is not forgetful.--> +<!-- It's similar to eager coercions ... keep all types around and error--> +<!-- if there's a new type that doesn't match the old ones.--> +<!-- Also, that paper chooses not to let functions have intersection types,--> +<!-- which kind-of-avoids the questions ... but really the eagerness is key.--> + +<p><a href="https://dl.acm.org/citation.cfm?id=3009849"><em>Big Types in Little Runtime</em></a>, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their <em>transient</em> semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.</p> + +<p>That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value&rsquo;s set. When a mismatch occurs, the semantics <a href="https://www.ccs.neu.edu/home/types/resources/notes/transient-undefined-blame-extract.pdf">tries to report</a> every type in the set that relates to the mismatch.</p> + +<p>And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">ICFP 2018</a>. Where will we see them next?</p> \ No newline at end of file diff --git a/blog/feeds/higher-order-contracts.rss.xml b/blog/feeds/higher-order-contracts.rss.xml new file mode 100644 index 00000000..b4a34f0c --- /dev/null +++ b/blog/feeds/higher-order-contracts.rss.xml @@ -0,0 +1,186 @@ + + + + PRL Blog: Posts tagged 'higher-order contracts' + PRL Blog: Posts tagged 'higher-order contracts' + http://prl.ccs.neu.edu/blog/tags/higher-order-contracts.html + Sun, 07 Apr 2019 23:15:11 UT + Sun, 07 Apr 2019 23:15:11 UT + 1800 + + Forgetful and Heedful contracts + http://prl.ccs.neu.edu/blog/2019/04/07/forgetful-and-heedful-contracts/?utm_source=higher-order-contracts&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-04-07-forgetful-and-heedful-contracts + Sun, 07 Apr 2019 23:15:11 UT + Ben Greenman + +<p><em>Forgetful</em> and <em>heedful</em> are two methods for space-efficient contracts developed by <a href="http://www.cs.pomona.edu/~michael/">Michael Greenberg</a> in <a href="https://arxiv.org/abs/1410.2813">2014</a>. These methods were born in the shadow of a third method, <em>eidetic</em>, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.</p> +<!-- more--> + +<hr /> + +<p>Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of <a href="https://www2.ccs.neu.edu/racket/pubs/popl11-dfff.pdf">three parties</a>: the API code, the client code, and the contract between them.</p> + +<p>For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:</p> + +<pre><code> #lang racket/base + + (module sort racket + (provide + (contract-out + [quicksort + (-&gt; (vectorof point/c) void?)])) + + (define point/c (vectorof integer?)) + + (define (quicksort points) + ....)) + + (module client racket + (require (submod ".." sort)) + (quicksort '())) + + (require 'client)</code></pre> + +<pre><code>quicksort: contract violation; + expected a vector + given: '() + in: the 1st argument of + (-&gt; (vectorof (vectorof integer?)) void?) + contract from: + (file.rkt sort) + blaming: (file.rkt client) + (assuming the contract is correct)</code></pre> + +<p>That covers the basics. For an extended introduction to contracts, visit <a href="https://docs.racket-lang.org/guide/contracts.html">The Racket Guide</a>.</p> + +<p>The quicksort example and the related figures are from the paper <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts: Fixing a Pathology of Gradual Typing</em></a></p> + +<h3 id="classic-contracts-and-space-efficiency">Classic contracts and &ldquo;Space Efficiency&rdquo;</h3> + +<p>The <code>(vectorof point/c)</code> contract used above describes a possibly-mutable array whose elements match the <code>point/c</code> contract. Since the array can be mutated, this contract has implications for two parties:</p> + +<ol> + <li>the client module must supply a good array, and</li> + <li>the sorting module must not insert a bad element.</li></ol> + +<p>To enforce the second condition, the <code>vectorof</code> contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:</p> + +<pre><code>(quicksort (vector (vector 4 4) + (vector 2 2) + (vector 1 1) + (vector 3 3)))</code></pre> + +<p>After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a <strong>solid</strong> blue bar for the <strong>write</strong> checks against the sort module and a <em>striped</em> blue bar for the <em>read</em> checks against the client.</p> + +<p><img src="/img/vector-chaperone-0.png" alt="A wrapped vector" /></p> + +<p>In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted &mdash; because every <strong>write</strong> and <em>read</em> applies a contract to the element.</p> + +<p><img src="/img/vector-chaperone-1.png" alt="Layers of element wrappers" /></p> + +<p>On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.</p> + +<p>Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and &ldquo;indirecting&rdquo; future writes/reads through wrappers. These space and time costs can add up.</p> + +<blockquote> + <p>&ldquo;on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times&rdquo; &mdash; <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts</em></a></p></blockquote> + +<p>To fix the problem, researchers have been exploring <em>space-efficient</em> implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and <em>eidetic</em>, <em>forgetful</em>, and <em>heedful</em> are his names for three implementations.</p> + +<p>(Although the goal of this post is to promote <em>forgetful</em> and <em>heedful</em>, we will review all three.)</p> + +<h3 id="eidetic-space-efficiency">Eidetic space-efficiency</h3> + +<p>The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a <em>merge</em> operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal &ldquo;wrapping&rdquo; contracts and their size is bounded by the number (and height) of source-code contracts in the program.</p> + +<p>An eidetic contract is an <code>N</code>-ary tree (for <code>N &gt; 0</code>):</p> + +<ul> + <li>each node represents a higher-order contract combinator, such as <code>vectorof</code></li> + <li>the <code>N</code> children of a node represent the different interactions that the value supports</li> + <li>each leaf is a list of non-higher-order, or <em>flat</em>, contracts</li></ul> + +<p>For example, the <code>(vectorof point/c)</code> source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf">Collapsible Contracts</a> paper has an illustration. Each tree node represents a <code>vectorof</code> contract; these nodes have <code>N=2</code> children because vectors support reads and writes.</p> + +<p>A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists &mdash; if an implementation comes with a <code>contract-stronger?</code> predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.</p> + +<p>Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should <code>contract-stronger?</code> try to solve problems that lack polynomial-time solutions?</p> + +<p>Thankfully, there are at least two &ldquo;compromise&rdquo; alternatives.</p> + +<h3 id="forgetful-space-efficiency">Forgetful space-efficiency</h3> +<!-- "no operation relies on e being a T2, skipping the check doesn't risk soundness" p.12--> +<!-- "In forgetful \lambda_H, we offer a simple solution to space inefficient casts: just forget about them" p.11--> +<!-- "Just the same, when accumulating casts on the stack, we throw away all but the last cast" p.11--> +<!-- "forgetful ... skip[s] checks and change[s] blame labels" p.3--> + +<blockquote> + <p>&ldquo;Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 10, bottom of page 23--></blockquote> + +<p>The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:</p> + +<ol> + <li>the client obligations from the old contract, and</li> + <li>the server obligations from the new contract</li></ol> + +<p>If not, proceed as usual &mdash; by wrapping (an unwrapped value) or raising an error. Every value receives at most <strong>one</strong> wrapper; this wrapper changes as the value flows to different clients.</p> + +<p>Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module <code>A</code> exports a function <code>f</code> with contract <code>(-&gt; T1 T2)</code> to module <code>B</code>, and suppose module <code>B</code> shares this function with a few other client modules using different contracts. As <code>f</code> flows to a new client, it keeps the <code>T1</code> domain check and gets a replacement for the <code>T2</code> codomain check.</p> + +<ul> + <li>Keeping <code>T1</code> ensures that the code inside the function (defined by module <code>A</code>) receives input that matches its expectation.</li> + <li>Replacing <code>T2</code> ensures that each new client receives output that it expects.</li></ul> + +<p>Unfortunately, replacing <code>T2</code> also means that clients of module <code>B</code> cannot trust the <code>T2</code> contract. This contract is not checked, and so forgetful contracts <strong>miss</strong> some errors that would be caught by standard contracts. For the same reason, a bug in module <code>B</code> may go undetected by its clients &mdash; even if a later contract reports an issue, the contract system has no memory that <code>B</code> was partly-responsible.</p> + +<p>Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.</p> + +<h3 id="heedful-space-efficiency">Heedful space-efficiency</h3> + +<p>A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set &mdash; or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.</p> + +<p>To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.</p> + +<p>The heedful method is a compromise between forgetful and eidetic.</p> + +<ul> + <li> + <p>Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of <code>contract-stronger?</code> predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.</p></li> + <li> + <p>Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.</p></li></ul> + +<p>For details, see <a href="https://arxiv.org/abs/1410.2813">the extended version</a> of Michael&rsquo;s POPL 2015 paper. Don&rsquo;t bother searching <a href="http://www.cs.pomona.edu/~michael/papers/popl2015_space.pdf">the conference version</a> &mdash; aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.</p> + +<p><code>*</code> If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to <a href="http://mballantyne.net/">Michael Ballantyne</a> for explaining this to me.</p> + +<h3 id="priorities-and-appearances">Priorities and Appearances</h3> + +<p>The extended version of <em>Space-Efficient Manifest Contracts</em> introduces the forgetful and heedful methods with extreme modesty. It&rsquo;s tempting to skip past them and focus on the eidetic method.</p> + +<blockquote> + <p>&ldquo;Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [<a href="https://dl.acm.org/citation.cfm?doid=1706299.1706342">Siek and Wadler 2010</a>]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen&mdash;warm up exercises for eidetic.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 1, bottom of page 2--></blockquote> + +<p>And yet, at least two other research papers rely on these &ldquo;strawmen&rdquo; &mdash; or rather, the ideas behind the names.</p> + +<p><a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a>, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.</p> + +<blockquote> + <p>&ldquo;if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- page 21--></blockquote> + +<p>This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.</p> + +<blockquote> + <p>&ldquo;removing these casts preserves the soundness of the evaluation while reducing the number of them&rdquo;</p> + <p>&ldquo;while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- p.27--><!-- page 21--></blockquote> +<!-- The followup at POPL 2019 is not forgetful.--> +<!-- It's similar to eager coercions ... keep all types around and error--> +<!-- if there's a new type that doesn't match the old ones.--> +<!-- Also, that paper chooses not to let functions have intersection types,--> +<!-- which kind-of-avoids the questions ... but really the eagerness is key.--> + +<p><a href="https://dl.acm.org/citation.cfm?id=3009849"><em>Big Types in Little Runtime</em></a>, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their <em>transient</em> semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.</p> + +<p>That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value&rsquo;s set. When a mismatch occurs, the semantics <a href="https://www.ccs.neu.edu/home/types/resources/notes/transient-undefined-blame-extract.pdf">tries to report</a> every type in the set that relates to the mismatch.</p> + +<p>And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">ICFP 2018</a>. Where will we see them next?</p> \ No newline at end of file diff --git a/blog/feeds/history.atom.xml b/blog/feeds/history.atom.xml new file mode 100644 index 00000000..754a2c2d --- /dev/null +++ b/blog/feeds/history.atom.xml @@ -0,0 +1,282 @@ + + + PRL Blog: Posts tagged 'history' + + + urn:http-prl-ccs-neu-edu:-blog-tags-history-html + 2019-09-05T10:00:00Z + + Lexical and Dynamic Scope + + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + 2019-09-05T10:00:00Z + 2019-09-05T10:00:00Z + + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> + + Trees, 1973 + + urn:http-prl-ccs-neu-edu:-blog-2017-07-19-trees-1973 + 2017-07-19T21:48:56Z + 2017-07-19T21:48:56Z + + Ben Greenman + +<p>From the PRL archives:</p> + +<blockquote> + <p>I think that I shall never see a matrix lovely as a tree. &mdash; <a href="/img/gls-trees-poem-1979.pdf"><em>Trees</em></a>, by Guy L. Steele Jr., MIT, 1973</p></blockquote> +<!-- more--> + +<hr /> + +<p>You might recognize the opening line from Joyce Kilmer&rsquo;s 1914 poem <a href="https://en.wikipedia.org/wiki/Trees_(poem)"><em>Trees</em></a>, or from Radia Perlman&rsquo;s <a href="/img/p-sigcomm-1985.pdf"><em>Algorhyme</em></a> (published 1985).</p> + +<p>The poem is online in <a href="http://mercury.lcs.mit.edu/~jnc/humour/lisp.tree">at least one other place</a>, but the copy linked above (from <a href="https://archive.org/details/byte-magazine">BYTE magazine</a>) comes with a footnote on <em>How this poem came to be printed</em>.</p> + + Continuations + + urn:http-prl-ccs-neu-edu:-blog-2017-07-17-continuations + 2017-07-17T12:52:07Z + 2017-07-17T12:52:07Z + + Ben Greenman + +<p>From the PRL archives:</p> + +<blockquote> + <p>It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. &mdash; <a href="/img/nall-continuations-1983.pdf"><em>Continuations</em></a> by Alan Nall, Indiana University, 1983</p></blockquote> +<!-- more--> + +<hr /> + +<p>I first encountered this essay on continuations in a green folder in the PRL. It turns out, the author spent a semester at Indiana University working on the <a href="http://wiki.c2.com/?SameFringeProblem">same fringe problem</a> for a graduate-level programming languages course. According to <a href="https://www.cs.indiana.edu/~dfried/">the instructor</a>: &ldquo;What he said was true. He could not stop thinking about the problem the entire semester.&rdquo; This essay was a kind of final exam.</p> + + Meaningful Distinctions + + urn:http-prl-ccs-neu-edu:-blog-2016-10-31-meaningful-distinctions + 2016-10-31T17:20:33Z + 2016-10-31T17:20:33Z + + Ben Greenman + +<blockquote> + <p>&ldquo;Meaningful distinctions deserve to be maintained.&rdquo; &mdash; Errett A. Bishop</p></blockquote> + +<p>Likewise, memorable quotations deserve to be read in context. In this spirit, I am happy to present the above &ldquo;basic principle&rdquo; in its context: <a href="/img/sicm.pdf"><em>Schizophrenia in contemporary mathematics</em> (pdf)</a></p> + +<p>Read on for a brief summary.</p> +<!-- more--> + +<hr /> + +<p>I first read the above quotation in <a href="http://www.michaelbeeson.com/research/papers/BishopForeword.pdf">Michael Beeson&rsquo;s introduction</a> to the 2012 edition of Bishop&rsquo;s <a href="https://www.amazon.com/Foundations-Constructive-Analysis-Errett-Bishop/dp/4871877140"><em>Foundations of Constructive Analysis</em></a>. That was two years ago.</p> + +<p>Last month, I tried to find its context. <a href="https://books.google.com/books?id=uPx8tGCaxzUC&amp;pg=PA214&amp;lpg=PA214&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=cWjwOTnNuT&amp;sig=wN143wNyfXtMFLGABBQM-22aSOQ&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIHjAA#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">Many</a> <a href="https://www.jstor.org/stable/2589553">other</a> <a href="https://books.google.com/books?id=J4DkBwAAQBAJ&amp;pg=PA6&amp;lpg=PA6&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=KYkrkBrJd_&amp;sig=AAK1A_uIkQlVcYCY1TFljfA3CqA&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIJTAC#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">uses</a> <a href="https://books.google.com/books?id=oN5nsPkXhhsC&amp;pg=PR6&amp;lpg=PR6&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=4doTufVdsy&amp;sig=u3e_Z_xdN-tjt9p1eqQ88juA5Ns&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIKDAD#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">of</a> <a href="https://books.google.com/books?id=GR44SKXCZJsC&amp;pg=RA1-PA199&amp;lpg=RA1-PA199&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=lNpzR5QV7h&amp;sig=IGg2Q_KtreSAhrbSJxsV7mQ8xok&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIMDAF#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">the</a> <a href="http://www.math.canterbury.ac.nz/php/groups/cm/faq/">quote</a> <a href="http://www.ben-sherman.net/aux/evident-logic.pdf">cited</a> a <em>Schizophrenia in comtemporary mathematics</em>, but I could not find an electronic copy. (It turns out, the AMS Bookstore page for <a href="http://bookstore.ams.org/conm-39"><em>Erret Bishop: Reflections on Him and His Research</em></a> includes a facsimile.)</p> + +<p>Lest anyone else be tempted to conjure the ancient magic of inter-library loan, here is a scan of the pages I borrowed. Thanks to the University of Toledo for supplying the hard copy.</p> + +<blockquote> + <p> <a href="/img/sicm.pdf">prl.ccs.neu.edu/img/sicm.pdf</a></p></blockquote> + +<p>The document is Bishop&rsquo;s &ldquo;feeling for the philosophical issues involved&rdquo; in constructive mathematics. First, Bishop lists &ldquo;schizophrenic attributes&rdquo; (trouble spots) of contemporary mathematics. Next, he gives basic principles of constructivism and Brouwer&rsquo;s interpretation of the logical quantifiers. Along the way, and as a source of examples, Bishop describes integers, sets, and real numbers. The emphasis is always on common-sense meaning and finite constructions.</p> + +<p>After a brief summary and reflection, the last ten pages list recent advances in constructive mathematics and upcoming tasks. The open tasks are particularly interesting:</p> + +<ul> + <li>systematically develop (constructive) algebra</li> + <li>give a constructive foundation for general topology</li> + <li>engage with the deeper &ldquo;meaning of mathematics&rdquo;</li></ul> + +<p>The popular quote on &ldquo;Meaningful Distinctions&rdquo; appears early in the paper, as one of Bishop&rsquo;s four principles that &ldquo;stand out as basic&rdquo; to the philosophy of constructivism:</p> + +<blockquote> + <p>A. Mathematics is common sense.</p> + <p>B. Do not ask whether a statement is true until you know what it means.</p> + <p>C. A proof is any completely convincing argument.</p> + <p>D. Meaningful distinctions deserve to be maintained.</p></blockquote> + +<p>I had no idea that D was &ldquo;a principle&rdquo;, or that it had three siblings.</p> + +<p>To further tempt you into reading the whole truth, here are some of my favorite phrases:</p> + +<blockquote> + <ul> + <li>One suspects that the majority of pure mathematicians &hellip; ignore as much content as they possibly can.</li> + <li>We have geared ourselves to producing research mathematicians who will begin to write papers as soon as possible. This anti-social and anti-intellectual process defeats even its own narrow ends.</li> + <li>&hellip; truth is not a source of trouble to the constructivist, because of his emphasis on meaning.</li> + <li>&hellip; guided primarily by considerations of content rather than elegance and formal attractiveness &hellip;</li> + <li>Let me tell you what a smart sequence will do.</li> + <li>Classical mathematics fails to observe meaningful distinctions having to do with integers.</li> + <li>Constructive mathematics does not postulate a pre-existent universe, with objects lying around waiting to be collected and grouped into sets, like shells on a beach.</li> + <li>It might be worthwhile to investigate the possibility that constructive mathematics would afford a solid philosophical basis for the theory of computation &hellip;</li> + <li>&hellip; if the product of two real numbers is 0, we are not entitled to conclude that one of them is 0.</li> + <li>It is fair to say that almost nobody finds his proof intelligible.</li> + <li>Mathematics is such a complicated activity that disagreements are bound to arise.</li> + <li>Algebraic topology, at least on the elementary level, should not be too difficult to constructivize.</li> + <li>I hope all this accords with your common sense, as it does with mine.</li></ul></blockquote> + +<p>Now go find their context!</p> + + History of Actors + + urn:http-prl-ccs-neu-edu:-blog-2016-10-19-history-of-actors + 2016-10-19T17:26:16Z + 2016-10-19T17:26:16Z + + Tony Garnock-Jones + +<p>Christos Dimoulas is currently teaching a <a href="http://www.seas.harvard.edu/courses/cs252/2016fa/">&ldquo;History of Programming Languages&rdquo; class at Harvard</a>. The class is, as Christos writes, &ldquo;definitely not about <a href="https://www.levenez.com/lang/lang_letter.pdf">this</a>&rdquo;; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.</p> + +<p>On Monday, I presented &ldquo;the History of Actors&rdquo; for the class. I&rsquo;ve made the written-out talk notes and an annotated bibliography available <a href="https://eighty-twenty.org/2016/10/18/actors-hopl">here</a>.</p> \ No newline at end of file diff --git a/blog/feeds/history.rss.xml b/blog/feeds/history.rss.xml new file mode 100644 index 00000000..929a9ad8 --- /dev/null +++ b/blog/feeds/history.rss.xml @@ -0,0 +1,274 @@ + + + + PRL Blog: Posts tagged 'history' + PRL Blog: Posts tagged 'history' + http://prl.ccs.neu.edu/blog/tags/history.html + Thu, 05 Sep 2019 10:00:00 UT + Thu, 05 Sep 2019 10:00:00 UT + 1800 + + Lexical and Dynamic Scope + http://prl.ccs.neu.edu/blog/2019/09/05/lexical-and-dynamic-scope/?utm_source=history&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + Thu, 05 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> + + Trees, 1973 + http://prl.ccs.neu.edu/blog/2017/07/19/trees-1973/?utm_source=history&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-07-19-trees-1973 + Wed, 19 Jul 2017 21:48:56 UT + Ben Greenman + +<p>From the PRL archives:</p> + +<blockquote> + <p>I think that I shall never see a matrix lovely as a tree. &mdash; <a href="/img/gls-trees-poem-1979.pdf"><em>Trees</em></a>, by Guy L. Steele Jr., MIT, 1973</p></blockquote> +<!-- more--> + +<hr /> + +<p>You might recognize the opening line from Joyce Kilmer&rsquo;s 1914 poem <a href="https://en.wikipedia.org/wiki/Trees_(poem)"><em>Trees</em></a>, or from Radia Perlman&rsquo;s <a href="/img/p-sigcomm-1985.pdf"><em>Algorhyme</em></a> (published 1985).</p> + +<p>The poem is online in <a href="http://mercury.lcs.mit.edu/~jnc/humour/lisp.tree">at least one other place</a>, but the copy linked above (from <a href="https://archive.org/details/byte-magazine">BYTE magazine</a>) comes with a footnote on <em>How this poem came to be printed</em>.</p> + + Continuations + http://prl.ccs.neu.edu/blog/2017/07/17/continuations/?utm_source=history&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-07-17-continuations + Mon, 17 Jul 2017 12:52:07 UT + Ben Greenman + +<p>From the PRL archives:</p> + +<blockquote> + <p>It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. &mdash; <a href="/img/nall-continuations-1983.pdf"><em>Continuations</em></a> by Alan Nall, Indiana University, 1983</p></blockquote> +<!-- more--> + +<hr /> + +<p>I first encountered this essay on continuations in a green folder in the PRL. It turns out, the author spent a semester at Indiana University working on the <a href="http://wiki.c2.com/?SameFringeProblem">same fringe problem</a> for a graduate-level programming languages course. According to <a href="https://www.cs.indiana.edu/~dfried/">the instructor</a>: &ldquo;What he said was true. He could not stop thinking about the problem the entire semester.&rdquo; This essay was a kind of final exam.</p> + + Meaningful Distinctions + http://prl.ccs.neu.edu/blog/2016/10/31/meaningful-distinctions/?utm_source=history&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-31-meaningful-distinctions + Mon, 31 Oct 2016 17:20:33 UT + Ben Greenman + +<blockquote> + <p>&ldquo;Meaningful distinctions deserve to be maintained.&rdquo; &mdash; Errett A. Bishop</p></blockquote> + +<p>Likewise, memorable quotations deserve to be read in context. In this spirit, I am happy to present the above &ldquo;basic principle&rdquo; in its context: <a href="/img/sicm.pdf"><em>Schizophrenia in contemporary mathematics</em> (pdf)</a></p> + +<p>Read on for a brief summary.</p> +<!-- more--> + +<hr /> + +<p>I first read the above quotation in <a href="http://www.michaelbeeson.com/research/papers/BishopForeword.pdf">Michael Beeson&rsquo;s introduction</a> to the 2012 edition of Bishop&rsquo;s <a href="https://www.amazon.com/Foundations-Constructive-Analysis-Errett-Bishop/dp/4871877140"><em>Foundations of Constructive Analysis</em></a>. That was two years ago.</p> + +<p>Last month, I tried to find its context. <a href="https://books.google.com/books?id=uPx8tGCaxzUC&amp;pg=PA214&amp;lpg=PA214&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=cWjwOTnNuT&amp;sig=wN143wNyfXtMFLGABBQM-22aSOQ&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIHjAA#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">Many</a> <a href="https://www.jstor.org/stable/2589553">other</a> <a href="https://books.google.com/books?id=J4DkBwAAQBAJ&amp;pg=PA6&amp;lpg=PA6&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=KYkrkBrJd_&amp;sig=AAK1A_uIkQlVcYCY1TFljfA3CqA&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIJTAC#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">uses</a> <a href="https://books.google.com/books?id=oN5nsPkXhhsC&amp;pg=PR6&amp;lpg=PR6&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=4doTufVdsy&amp;sig=u3e_Z_xdN-tjt9p1eqQ88juA5Ns&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIKDAD#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">of</a> <a href="https://books.google.com/books?id=GR44SKXCZJsC&amp;pg=RA1-PA199&amp;lpg=RA1-PA199&amp;dq=meaningful+distinctions+deserve+to+be+maintained&amp;source=bl&amp;ots=lNpzR5QV7h&amp;sig=IGg2Q_KtreSAhrbSJxsV7mQ8xok&amp;hl=en&amp;sa=X&amp;ved=0ahUKEwjPwt2HmYbQAhWE6IMKHU5rB8YQ6AEIMDAF#v=onepage&amp;q=meaningful%20distinctions%20deserve%20to%20be%20maintained&amp;f=false">the</a> <a href="http://www.math.canterbury.ac.nz/php/groups/cm/faq/">quote</a> <a href="http://www.ben-sherman.net/aux/evident-logic.pdf">cited</a> a <em>Schizophrenia in comtemporary mathematics</em>, but I could not find an electronic copy. (It turns out, the AMS Bookstore page for <a href="http://bookstore.ams.org/conm-39"><em>Erret Bishop: Reflections on Him and His Research</em></a> includes a facsimile.)</p> + +<p>Lest anyone else be tempted to conjure the ancient magic of inter-library loan, here is a scan of the pages I borrowed. Thanks to the University of Toledo for supplying the hard copy.</p> + +<blockquote> + <p> <a href="/img/sicm.pdf">prl.ccs.neu.edu/img/sicm.pdf</a></p></blockquote> + +<p>The document is Bishop&rsquo;s &ldquo;feeling for the philosophical issues involved&rdquo; in constructive mathematics. First, Bishop lists &ldquo;schizophrenic attributes&rdquo; (trouble spots) of contemporary mathematics. Next, he gives basic principles of constructivism and Brouwer&rsquo;s interpretation of the logical quantifiers. Along the way, and as a source of examples, Bishop describes integers, sets, and real numbers. The emphasis is always on common-sense meaning and finite constructions.</p> + +<p>After a brief summary and reflection, the last ten pages list recent advances in constructive mathematics and upcoming tasks. The open tasks are particularly interesting:</p> + +<ul> + <li>systematically develop (constructive) algebra</li> + <li>give a constructive foundation for general topology</li> + <li>engage with the deeper &ldquo;meaning of mathematics&rdquo;</li></ul> + +<p>The popular quote on &ldquo;Meaningful Distinctions&rdquo; appears early in the paper, as one of Bishop&rsquo;s four principles that &ldquo;stand out as basic&rdquo; to the philosophy of constructivism:</p> + +<blockquote> + <p>A. Mathematics is common sense.</p> + <p>B. Do not ask whether a statement is true until you know what it means.</p> + <p>C. A proof is any completely convincing argument.</p> + <p>D. Meaningful distinctions deserve to be maintained.</p></blockquote> + +<p>I had no idea that D was &ldquo;a principle&rdquo;, or that it had three siblings.</p> + +<p>To further tempt you into reading the whole truth, here are some of my favorite phrases:</p> + +<blockquote> + <ul> + <li>One suspects that the majority of pure mathematicians &hellip; ignore as much content as they possibly can.</li> + <li>We have geared ourselves to producing research mathematicians who will begin to write papers as soon as possible. This anti-social and anti-intellectual process defeats even its own narrow ends.</li> + <li>&hellip; truth is not a source of trouble to the constructivist, because of his emphasis on meaning.</li> + <li>&hellip; guided primarily by considerations of content rather than elegance and formal attractiveness &hellip;</li> + <li>Let me tell you what a smart sequence will do.</li> + <li>Classical mathematics fails to observe meaningful distinctions having to do with integers.</li> + <li>Constructive mathematics does not postulate a pre-existent universe, with objects lying around waiting to be collected and grouped into sets, like shells on a beach.</li> + <li>It might be worthwhile to investigate the possibility that constructive mathematics would afford a solid philosophical basis for the theory of computation &hellip;</li> + <li>&hellip; if the product of two real numbers is 0, we are not entitled to conclude that one of them is 0.</li> + <li>It is fair to say that almost nobody finds his proof intelligible.</li> + <li>Mathematics is such a complicated activity that disagreements are bound to arise.</li> + <li>Algebraic topology, at least on the elementary level, should not be too difficult to constructivize.</li> + <li>I hope all this accords with your common sense, as it does with mine.</li></ul></blockquote> + +<p>Now go find their context!</p> + + History of Actors + http://prl.ccs.neu.edu/blog/2016/10/19/history-of-actors/?utm_source=history&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-19-history-of-actors + Wed, 19 Oct 2016 17:26:16 UT + Tony Garnock-Jones + +<p>Christos Dimoulas is currently teaching a <a href="http://www.seas.harvard.edu/courses/cs252/2016fa/">&ldquo;History of Programming Languages&rdquo; class at Harvard</a>. The class is, as Christos writes, &ldquo;definitely not about <a href="https://www.levenez.com/lang/lang_letter.pdf">this</a>&rdquo;; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.</p> + +<p>On Monday, I presented &ldquo;the History of Actors&rdquo; for the class. I&rsquo;ve made the written-out talk notes and an annotated bibliography available <a href="https://eighty-twenty.org/2016/10/18/actors-hopl">here</a>.</p> \ No newline at end of file diff --git a/blog/feeds/icfp.atom.xml b/blog/feeds/icfp.atom.xml new file mode 100644 index 00000000..6d41cde9 --- /dev/null +++ b/blog/feeds/icfp.atom.xml @@ -0,0 +1,85 @@ + + + PRL Blog: Posts tagged 'icfp' + + + urn:http-prl-ccs-neu-edu:-blog-tags-icfp-html + 2016-11-16T00:00:00Z + + Understanding Constructive Galois Connections + + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + 2016-11-16T00:00:00Z + 2016-11-16T00:00:00Z + + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/icfp.rss.xml b/blog/feeds/icfp.rss.xml new file mode 100644 index 00000000..4bf6d6c6 --- /dev/null +++ b/blog/feeds/icfp.rss.xml @@ -0,0 +1,85 @@ + + + + PRL Blog: Posts tagged 'icfp' + PRL Blog: Posts tagged 'icfp' + http://prl.ccs.neu.edu/blog/tags/icfp.html + Wed, 16 Nov 2016 00:00:00 UT + Wed, 16 Nov 2016 00:00:00 UT + 1800 + + Understanding Constructive Galois Connections + http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/?utm_source=icfp&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + Wed, 16 Nov 2016 00:00:00 UT + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/instrumentation.atom.xml b/blog/feeds/instrumentation.atom.xml new file mode 100644 index 00000000..ebff19ee --- /dev/null +++ b/blog/feeds/instrumentation.atom.xml @@ -0,0 +1,456 @@ + + + PRL Blog: Posts tagged 'instrumentation' + + + urn:http-prl-ccs-neu-edu:-blog-tags-instrumentation-html + 2016-05-24T10:51:34Z + + Measuring GC latencies in Haskell, OCaml, Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + 2016-05-24T10:51:34Z + 2016-05-24T10:51:34Z + + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/instrumentation.rss.xml b/blog/feeds/instrumentation.rss.xml new file mode 100644 index 00000000..f9a9b41b --- /dev/null +++ b/blog/feeds/instrumentation.rss.xml @@ -0,0 +1,456 @@ + + + + PRL Blog: Posts tagged 'instrumentation' + PRL Blog: Posts tagged 'instrumentation' + http://prl.ccs.neu.edu/blog/tags/instrumentation.html + Tue, 24 May 2016 10:51:34 UT + Tue, 24 May 2016 10:51:34 UT + 1800 + + Measuring GC latencies in Haskell, OCaml, Racket + http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/?utm_source=instrumentation&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + Tue, 24 May 2016 10:51:34 UT + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/interactive-syntax.atom.xml b/blog/feeds/interactive-syntax.atom.xml new file mode 100644 index 00000000..f3f74275 --- /dev/null +++ b/blog/feeds/interactive-syntax.atom.xml @@ -0,0 +1,252 @@ + + + PRL Blog: Posts tagged 'interactive syntax' + + + urn:http-prl-ccs-neu-edu:-blog-tags-interactive-syntax-html + 2022-01-06T17:56:08Z + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + 2022-01-06T17:56:08Z + 2022-01-06T17:56:08Z + + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> \ No newline at end of file diff --git a/blog/feeds/interactive-syntax.rss.xml b/blog/feeds/interactive-syntax.rss.xml new file mode 100644 index 00000000..d1e9448f --- /dev/null +++ b/blog/feeds/interactive-syntax.rss.xml @@ -0,0 +1,252 @@ + + + + PRL Blog: Posts tagged 'interactive syntax' + PRL Blog: Posts tagged 'interactive syntax' + http://prl.ccs.neu.edu/blog/tags/interactive-syntax.html + Thu, 06 Jan 2022 17:56:08 UT + Thu, 06 Jan 2022 17:56:08 UT + 1800 + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + http://prl.ccs.neu.edu/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/?utm_source=interactive-syntax&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + Thu, 06 Jan 2022 17:56:08 UT + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> \ No newline at end of file diff --git a/blog/feeds/java.atom.xml b/blog/feeds/java.atom.xml new file mode 100644 index 00000000..8688dcd9 --- /dev/null +++ b/blog/feeds/java.atom.xml @@ -0,0 +1,303 @@ + + + PRL Blog: Posts tagged 'java' + + + urn:http-prl-ccs-neu-edu:-blog-tags-java-html + 2018-12-02T14:41:53Z + + Java and Migratory Typing + + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + 2018-12-02T14:41:53Z + 2018-12-02T14:41:53Z + + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> \ No newline at end of file diff --git a/blog/feeds/java.rss.xml b/blog/feeds/java.rss.xml new file mode 100644 index 00000000..f390599a --- /dev/null +++ b/blog/feeds/java.rss.xml @@ -0,0 +1,303 @@ + + + + PRL Blog: Posts tagged 'java' + PRL Blog: Posts tagged 'java' + http://prl.ccs.neu.edu/blog/tags/java.html + Sun, 02 Dec 2018 14:41:53 UT + Sun, 02 Dec 2018 14:41:53 UT + 1800 + + Java and Migratory Typing + http://prl.ccs.neu.edu/blog/2018/12/02/java-and-migratory-typing/?utm_source=java&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + Sun, 02 Dec 2018 14:41:53 UT + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> \ No newline at end of file diff --git a/blog/feeds/lambda.atom.xml b/blog/feeds/lambda.atom.xml new file mode 100644 index 00000000..b005b53a --- /dev/null +++ b/blog/feeds/lambda.atom.xml @@ -0,0 +1,306 @@ + + + PRL Blog: Posts tagged 'lambda' + + + urn:http-prl-ccs-neu-edu:-blog-tags-lambda-html + 2016-11-02T21:10:18Z + + Beta Reduction (Part 1) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + 2016-11-02T21:10:18Z + 2016-11-02T21:10:18Z + + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/lambda.rss.xml b/blog/feeds/lambda.rss.xml new file mode 100644 index 00000000..9a20288b --- /dev/null +++ b/blog/feeds/lambda.rss.xml @@ -0,0 +1,306 @@ + + + + PRL Blog: Posts tagged 'lambda' + PRL Blog: Posts tagged 'lambda' + http://prl.ccs.neu.edu/blog/tags/lambda.html + Wed, 02 Nov 2016 21:10:18 UT + Wed, 02 Nov 2016 21:10:18 UT + 1800 + + Beta Reduction (Part 1) + http://prl.ccs.neu.edu/blog/2016/11/02/beta-reduction-part-1/?utm_source=lambda&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + Wed, 02 Nov 2016 21:10:18 UT + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/lang-extension.atom.xml b/blog/feeds/lang-extension.atom.xml new file mode 100644 index 00000000..2db655dc --- /dev/null +++ b/blog/feeds/lang-extension.atom.xml @@ -0,0 +1,362 @@ + + + PRL Blog: Posts tagged 'lang-extension' + + + urn:http-prl-ccs-neu-edu:-blog-tags-lang-extension-html + 2017-03-03T08:54:20Z + + PLT Redex: mf-apply + + urn:http-prl-ccs-neu-edu:-blog-2017-03-03-plt-redex-mf-apply + 2017-03-03T08:54:20Z + 2017-03-03T08:54:20Z + + Ben Greenman + +<p>The <code>mf-apply</code> keyword is for checked metafunction application in PLT Redex. In other words, <code>(mf-apply f x)</code> is just like <code>(f x)</code>, but errors if <code>f</code> is not a previously-defined metafunction.</p> + +<p>Also, consider applying to attend <em>The Racket School of Semantics and Languages</em> in Salt Lake City this summer: <a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></p> +<!-- more--> + +<h2 id="metafunctions-vs-list-patterns">Metafunctions vs. List Patterns</h2> + +<p>Have you used PLT Redex? Good! Maybe this has happened to you:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span> +<span class="normal">25</span> +<span class="normal">26</span> +<span class="normal">27</span> +<span class="normal">28</span> +<span class="normal">29</span> +<span class="normal">30</span> +<span class="normal">31</span> +<span class="normal">32</span> +<span class="normal">33</span> +<span class="normal">34</span> +<span class="normal">35</span> +<span class="normal">36</span> +<span class="normal">37</span> +<span class="normal">38</span> +<span class="normal">39</span> +<span class="normal">40</span> +<span class="normal">41</span> +<span class="normal">42</span> +<span class="normal">43</span> +<span class="normal">44</span> +<span class="normal">45</span> +<span class="normal">46</span> +<span class="normal">47</span> +<span class="normal">48</span> +<span class="normal">49</span> +<span class="normal">50</span> +<span class="normal">51</span> +<span class="normal">52</span> +<span class="normal">53</span> +<span class="normal">54</span> +<span class="normal">55</span> +<span class="normal">56</span> +<span class="normal">57</span> +<span class="normal">58</span> +<span class="normal">59</span> +<span class="normal">60</span> +<span class="normal">61</span> +<span class="normal">62</span> +<span class="normal">63</span> +<span class="normal">64</span> +<span class="normal">65</span> +<span class="normal">66</span> +<span class="normal">67</span> +<span class="normal">68</span> +<span class="normal">69</span> +<span class="normal">70</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 1. You define a language</span> +<span class="p">(</span><span class="n">define-language</span> <span class="n">STLC</span> + <span class="p">[</span><span class="n">V</span> <span class="n">::=</span> <span class="n">integer</span> <span class="n">boolean</span> <span class="n">C</span><span class="p">]</span> + <span class="p">[</span><span class="n">C</span> <span class="n">::=</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">Λ</span> <span class="n">::=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)]</span> + <span class="p">[</span><span class="n">M</span> <span class="n">::=</span> <span class="p">(</span><span class="n">M</span> <span class="n">M</span><span class="p">)</span> <span class="n">V</span> <span class="n">Λ</span> <span class="n">x</span><span class="p">]</span> + <span class="p">[</span><span class="n">τ</span> <span class="n">::=</span> <span class="n">Int</span> <span class="n">Bool</span> <span class="p">(</span><span class="n">τ</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">ρ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">V</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">Γ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">τ</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">x</span> <span class="n">::=</span> <span class="n">variable-not-otherwise-mentioned</span><span class="p">]</span> + <span class="kd">#:binding-forms</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span> <span class="kd">#:refers-to</span> <span class="n">x</span><span class="p">))</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 2. You define a few metafunctions</span> +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;lam</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">Λ</span> + <span class="p">[(</span><span class="n">closure-&gt;lam</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">Λ</span><span class="p">])</span> + +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;env</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">ρ</span> + <span class="p">[(</span><span class="n">closure-&gt;env</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">ρ</span><span class="p">])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 3. You try defining a judgment form . . .</span> +<span class="p">(</span><span class="n">define-judgment-form</span> <span class="n">STLC</span> + <span class="kd">#:mode</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">I</span> <span class="n">O</span><span class="p">)</span> + <span class="kd">#:contract</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Var</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">x</span> <span class="p">(</span><span class="n">x</span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_0</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_1</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">---</span> <span class="n">FVS-App</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">M_0</span> <span class="n">M_1</span><span class="p">)</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x_0</span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)</span> <span class="n">Λ</span><span class="p">)</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-remove))" style="color: inherit">set-remove</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="n">x_0</span><span class="p">)))</span> + <span class="n">---</span> <span class="n">FVS-Λ</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Integer</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">integer_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Boolean</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">boolean_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">closure-&gt;lam</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-subtract))" style="color: inherit">set-subtract</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))))</span> + <span class="n">---</span> <span class="n">FVS-Closure</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">C</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 4. You test the judgment, and it mysteriously fails</span> +<span class="p">(</span><span class="n">judgment-holds</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">closure</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">())</span> + <span class="p">()))</span> +<span class="c1">;; ==&gt; #f</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><strong>WHAT HAPPENED??!</strong></p> + +<p>The problem is this line in the <code>FVS-Closure</code> rule:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which checks that the list <code>(closure-env C)</code> (whose first element is the symbol <code>closure-env</code> and second element is the symbol <code>C</code>) matches the pattern <code>((x_1 τ_1) ...)</code>.</p> + +<p>Right.</p> + +<p>Of course you meant to apply the metafunction <code>closure-&gt;env</code> but made a typo. And since the syntax for metafunction application is the same as the syntax for building a list, Redex doesn&rsquo;t report an error.</p> + +<p>We can fix this code with the new <a href="https://www.cs.utah.edu/plt/snapshots/current/doc/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._mf-apply%29%29"><code>mf-apply</code></a> keyword (available on <a href="https://github.com/racket/racket">GitHub</a> or in a <a href="https://www.cs.utah.edu/plt/snapshots/">snapshot build</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running <code>raco make</code> now gives a compile-time error.</p> + +<pre><code> term: expected a previously defined metafunction + at: closure-env + in: (mf-apply closure-env C)</code></pre> + +<h3 id="but-i-still-need-to-type-mf-apply-correctly">But I still need to type <code>mf-apply</code> correctly!</h3> + +<p>Leif Andersen says:</p> + +<blockquote> + <p>I should point out that this has the issue of you still need to type <code>mf-apply</code> correctly. ;)</p></blockquote> + +<p>That is, if you accidentally write:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-applu</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then the code compiles, thinking you intend to match a list of three elements against the pattern.</p> + +<p>Never fear, there are at least two solutions.</p> + +<h4 id="solution-1-rename-mf-apply">Solution 1: rename <code>mf-apply</code></h4> + +<p>A simple fix is to rename the <code>mf-apply</code> keyword to something shorter (and harder to mis-type):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._rename-in))" style="color: inherit">rename-in</a></span> <span class="n">redex</span> + <span class="p">[</span><span class="n">mf-apply</span> <span class="n">MF</span><span class="p">]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="solution-2-the-mf-apply-lang-extension">Solution 2: the <code>mf-apply</code> lang extension</h4> + +<p>A fancier solution is to install the <code>mf-apply</code> meta-language.</p> + +<pre><code> $ raco pkg install mf-apply</code></pre> + +<p>This language updates the <a href="http://docs.racket-lang.org/reference/readtables.html#%28tech._readtable%29"><em>readtable</em></a> to interpret S-expressions that begin with <code>#{</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="o">#</span><span class="p">{</span><span class="ss">f</span> <span class="ss">x</span> <span class="ss"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>as a metafunction application:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">f</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>You the programmer only needs to write the <code>#{....}</code> syntax.</p> + +<p>Source code is on GitHub:</p> + +<ul> + <li><a href="https://github.com/bennn/mf-apply">https://github.com/bennn/mf-apply</a></li></ul> + +<p>(It&rsquo;s the simplest lang-extension I know of)</p> + +<h2 id="what-is-plt-redex">What is PLT Redex?</h2> + +<p>PLT Redex is a library for semantics engineering. One of my favorite Redex features is it implements capture-avoiding substitution and α-equivalence for any language with a <code>#:binding-forms</code> specification (such as STLC, above).</p> + +<p>You can read more:</p> + +<ul> + <li>in the &ldquo;Amb&rdquo; tutorial: <a href="http://docs.racket-lang.org/redex/tutorial.html">http://docs.racket-lang.org/redex/tutorial.html</a></li> + <li>in the &ldquo;Long Tutorial&rdquo;: <a href="http://docs.racket-lang.org/redex/redex2015.html">http://docs.racket-lang.org/redex/redex2015.html</a></li> + <li>in the Redex reference manual: <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html">http://docs.racket-lang.org/redex/The_Redex_Reference.html</a></li> + <li>on the PLT Redex website: <a href="https://redex.racket-lang.org/">https://redex.racket-lang.org/</a></li> + <li>on GitHub: <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + +<p>And if you act now, you can become a <em>Redexan</em> between July 10 and July 14 at the summer school in Salt Lake City, Utah:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/lang-extension.rss.xml b/blog/feeds/lang-extension.rss.xml new file mode 100644 index 00000000..2408f806 --- /dev/null +++ b/blog/feeds/lang-extension.rss.xml @@ -0,0 +1,362 @@ + + + + PRL Blog: Posts tagged 'lang-extension' + PRL Blog: Posts tagged 'lang-extension' + http://prl.ccs.neu.edu/blog/tags/lang-extension.html + Fri, 03 Mar 2017 08:54:20 UT + Fri, 03 Mar 2017 08:54:20 UT + 1800 + + PLT Redex: mf-apply + http://prl.ccs.neu.edu/blog/2017/03/03/plt-redex-mf-apply/?utm_source=lang-extension&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-03-03-plt-redex-mf-apply + Fri, 03 Mar 2017 08:54:20 UT + Ben Greenman + +<p>The <code>mf-apply</code> keyword is for checked metafunction application in PLT Redex. In other words, <code>(mf-apply f x)</code> is just like <code>(f x)</code>, but errors if <code>f</code> is not a previously-defined metafunction.</p> + +<p>Also, consider applying to attend <em>The Racket School of Semantics and Languages</em> in Salt Lake City this summer: <a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></p> +<!-- more--> + +<h2 id="metafunctions-vs-list-patterns">Metafunctions vs. List Patterns</h2> + +<p>Have you used PLT Redex? Good! Maybe this has happened to you:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span> +<span class="normal">25</span> +<span class="normal">26</span> +<span class="normal">27</span> +<span class="normal">28</span> +<span class="normal">29</span> +<span class="normal">30</span> +<span class="normal">31</span> +<span class="normal">32</span> +<span class="normal">33</span> +<span class="normal">34</span> +<span class="normal">35</span> +<span class="normal">36</span> +<span class="normal">37</span> +<span class="normal">38</span> +<span class="normal">39</span> +<span class="normal">40</span> +<span class="normal">41</span> +<span class="normal">42</span> +<span class="normal">43</span> +<span class="normal">44</span> +<span class="normal">45</span> +<span class="normal">46</span> +<span class="normal">47</span> +<span class="normal">48</span> +<span class="normal">49</span> +<span class="normal">50</span> +<span class="normal">51</span> +<span class="normal">52</span> +<span class="normal">53</span> +<span class="normal">54</span> +<span class="normal">55</span> +<span class="normal">56</span> +<span class="normal">57</span> +<span class="normal">58</span> +<span class="normal">59</span> +<span class="normal">60</span> +<span class="normal">61</span> +<span class="normal">62</span> +<span class="normal">63</span> +<span class="normal">64</span> +<span class="normal">65</span> +<span class="normal">66</span> +<span class="normal">67</span> +<span class="normal">68</span> +<span class="normal">69</span> +<span class="normal">70</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 1. You define a language</span> +<span class="p">(</span><span class="n">define-language</span> <span class="n">STLC</span> + <span class="p">[</span><span class="n">V</span> <span class="n">::=</span> <span class="n">integer</span> <span class="n">boolean</span> <span class="n">C</span><span class="p">]</span> + <span class="p">[</span><span class="n">C</span> <span class="n">::=</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">Λ</span> <span class="n">::=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)]</span> + <span class="p">[</span><span class="n">M</span> <span class="n">::=</span> <span class="p">(</span><span class="n">M</span> <span class="n">M</span><span class="p">)</span> <span class="n">V</span> <span class="n">Λ</span> <span class="n">x</span><span class="p">]</span> + <span class="p">[</span><span class="n">τ</span> <span class="n">::=</span> <span class="n">Int</span> <span class="n">Bool</span> <span class="p">(</span><span class="n">τ</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">ρ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">V</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">Γ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">τ</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">x</span> <span class="n">::=</span> <span class="n">variable-not-otherwise-mentioned</span><span class="p">]</span> + <span class="kd">#:binding-forms</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span> <span class="kd">#:refers-to</span> <span class="n">x</span><span class="p">))</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 2. You define a few metafunctions</span> +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;lam</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">Λ</span> + <span class="p">[(</span><span class="n">closure-&gt;lam</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">Λ</span><span class="p">])</span> + +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;env</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">ρ</span> + <span class="p">[(</span><span class="n">closure-&gt;env</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">ρ</span><span class="p">])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 3. You try defining a judgment form . . .</span> +<span class="p">(</span><span class="n">define-judgment-form</span> <span class="n">STLC</span> + <span class="kd">#:mode</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">I</span> <span class="n">O</span><span class="p">)</span> + <span class="kd">#:contract</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Var</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">x</span> <span class="p">(</span><span class="n">x</span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_0</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_1</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">---</span> <span class="n">FVS-App</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">M_0</span> <span class="n">M_1</span><span class="p">)</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x_0</span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)</span> <span class="n">Λ</span><span class="p">)</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-remove))" style="color: inherit">set-remove</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="n">x_0</span><span class="p">)))</span> + <span class="n">---</span> <span class="n">FVS-Λ</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Integer</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">integer_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Boolean</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">boolean_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">closure-&gt;lam</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-subtract))" style="color: inherit">set-subtract</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))))</span> + <span class="n">---</span> <span class="n">FVS-Closure</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">C</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 4. You test the judgment, and it mysteriously fails</span> +<span class="p">(</span><span class="n">judgment-holds</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">closure</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">())</span> + <span class="p">()))</span> +<span class="c1">;; ==&gt; #f</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><strong>WHAT HAPPENED??!</strong></p> + +<p>The problem is this line in the <code>FVS-Closure</code> rule:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which checks that the list <code>(closure-env C)</code> (whose first element is the symbol <code>closure-env</code> and second element is the symbol <code>C</code>) matches the pattern <code>((x_1 τ_1) ...)</code>.</p> + +<p>Right.</p> + +<p>Of course you meant to apply the metafunction <code>closure-&gt;env</code> but made a typo. And since the syntax for metafunction application is the same as the syntax for building a list, Redex doesn&rsquo;t report an error.</p> + +<p>We can fix this code with the new <a href="https://www.cs.utah.edu/plt/snapshots/current/doc/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._mf-apply%29%29"><code>mf-apply</code></a> keyword (available on <a href="https://github.com/racket/racket">GitHub</a> or in a <a href="https://www.cs.utah.edu/plt/snapshots/">snapshot build</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running <code>raco make</code> now gives a compile-time error.</p> + +<pre><code> term: expected a previously defined metafunction + at: closure-env + in: (mf-apply closure-env C)</code></pre> + +<h3 id="but-i-still-need-to-type-mf-apply-correctly">But I still need to type <code>mf-apply</code> correctly!</h3> + +<p>Leif Andersen says:</p> + +<blockquote> + <p>I should point out that this has the issue of you still need to type <code>mf-apply</code> correctly. ;)</p></blockquote> + +<p>That is, if you accidentally write:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-applu</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then the code compiles, thinking you intend to match a list of three elements against the pattern.</p> + +<p>Never fear, there are at least two solutions.</p> + +<h4 id="solution-1-rename-mf-apply">Solution 1: rename <code>mf-apply</code></h4> + +<p>A simple fix is to rename the <code>mf-apply</code> keyword to something shorter (and harder to mis-type):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._rename-in))" style="color: inherit">rename-in</a></span> <span class="n">redex</span> + <span class="p">[</span><span class="n">mf-apply</span> <span class="n">MF</span><span class="p">]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="solution-2-the-mf-apply-lang-extension">Solution 2: the <code>mf-apply</code> lang extension</h4> + +<p>A fancier solution is to install the <code>mf-apply</code> meta-language.</p> + +<pre><code> $ raco pkg install mf-apply</code></pre> + +<p>This language updates the <a href="http://docs.racket-lang.org/reference/readtables.html#%28tech._readtable%29"><em>readtable</em></a> to interpret S-expressions that begin with <code>#{</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="o">#</span><span class="p">{</span><span class="ss">f</span> <span class="ss">x</span> <span class="ss"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>as a metafunction application:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">f</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>You the programmer only needs to write the <code>#{....}</code> syntax.</p> + +<p>Source code is on GitHub:</p> + +<ul> + <li><a href="https://github.com/bennn/mf-apply">https://github.com/bennn/mf-apply</a></li></ul> + +<p>(It&rsquo;s the simplest lang-extension I know of)</p> + +<h2 id="what-is-plt-redex">What is PLT Redex?</h2> + +<p>PLT Redex is a library for semantics engineering. One of my favorite Redex features is it implements capture-avoiding substitution and α-equivalence for any language with a <code>#:binding-forms</code> specification (such as STLC, above).</p> + +<p>You can read more:</p> + +<ul> + <li>in the &ldquo;Amb&rdquo; tutorial: <a href="http://docs.racket-lang.org/redex/tutorial.html">http://docs.racket-lang.org/redex/tutorial.html</a></li> + <li>in the &ldquo;Long Tutorial&rdquo;: <a href="http://docs.racket-lang.org/redex/redex2015.html">http://docs.racket-lang.org/redex/redex2015.html</a></li> + <li>in the Redex reference manual: <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html">http://docs.racket-lang.org/redex/The_Redex_Reference.html</a></li> + <li>on the PLT Redex website: <a href="https://redex.racket-lang.org/">https://redex.racket-lang.org/</a></li> + <li>on GitHub: <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + +<p>And if you act now, you can become a <em>Redexan</em> between July 10 and July 14 at the summer school in Salt Lake City, Utah:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/language-implementation.atom.xml b/blog/feeds/language-implementation.atom.xml new file mode 100644 index 00000000..4c7f44bc --- /dev/null +++ b/blog/feeds/language-implementation.atom.xml @@ -0,0 +1,83 @@ + + + PRL Blog: Posts tagged 'language implementation' + + + urn:http-prl-ccs-neu-edu:-blog-tags-language-implementation-html + 2019-03-09T14:40:16Z + + PLISS: Learn About PL Implementation in a Castle + + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + 2019-03-09T14:40:16Z + 2019-03-09T14:40:16Z + + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> + + PLISS: Oregon without the greek + + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-pliss-oregon-without-the-greek + 2017-02-28T23:01:00Z + 2017-02-28T23:01:00Z + + Jan Vitek + +<p>What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or <a href="https://pliss2017.github.io">PLISS</a> for short) has set out to answer.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2017.github.io/speakers.html">twelve speakers</a> versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.</p> + +<p>Lectures cover current research and future trends in programming language design and implementation, including:</p> + +<ul> + <li>Writing Just-in-time Compilers with LLVM with Jan Vitek</li> + <li>Performance Evaluation and Benchmarking with Laurie Tratt</li> + <li>Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller</li> + <li>High-Performance Fully Concurrent Garbage Collection with Richard Jones</li> + <li>Compiling Dynamic Languages with David Edelsohn</li> + <li>Language-support for Distributed Datastores with Suresh Jagannathan</li> + <li>Testing Programming Language Implementations with Alastair Donaldson</li> + <li>Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might</li> + <li>The evolution of Scala with Martin Odersky</li></ul> + +<p>A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.</p> + +<p>Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.</p> + +<p>More information at:</p> + +<p><a href="https://pliss2017.github.io">https://pliss2017.github.io</a></p> + +<p>(Oregon is a reference to the <a href="https://www.cs.uoregon.edu/research/summerschool/summer17/">OPLSS</a>, in which you may also be interested.)</p> \ No newline at end of file diff --git a/blog/feeds/language-implementation.rss.xml b/blog/feeds/language-implementation.rss.xml new file mode 100644 index 00000000..50fe3875 --- /dev/null +++ b/blog/feeds/language-implementation.rss.xml @@ -0,0 +1,81 @@ + + + + PRL Blog: Posts tagged 'language implementation' + PRL Blog: Posts tagged 'language implementation' + http://prl.ccs.neu.edu/blog/tags/language-implementation.html + Sat, 09 Mar 2019 14:40:16 UT + Sat, 09 Mar 2019 14:40:16 UT + 1800 + + PLISS: Learn About PL Implementation in a Castle + http://prl.ccs.neu.edu/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/?utm_source=language-implementation&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + Sat, 09 Mar 2019 14:40:16 UT + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> + + PLISS: Oregon without the greek + http://prl.ccs.neu.edu/blog/2017/02/28/pliss-oregon-without-the-greek/?utm_source=language-implementation&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-pliss-oregon-without-the-greek + Tue, 28 Feb 2017 23:01:00 UT + Jan Vitek + +<p>What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or <a href="https://pliss2017.github.io">PLISS</a> for short) has set out to answer.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2017.github.io/speakers.html">twelve speakers</a> versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.</p> + +<p>Lectures cover current research and future trends in programming language design and implementation, including:</p> + +<ul> + <li>Writing Just-in-time Compilers with LLVM with Jan Vitek</li> + <li>Performance Evaluation and Benchmarking with Laurie Tratt</li> + <li>Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller</li> + <li>High-Performance Fully Concurrent Garbage Collection with Richard Jones</li> + <li>Compiling Dynamic Languages with David Edelsohn</li> + <li>Language-support for Distributed Datastores with Suresh Jagannathan</li> + <li>Testing Programming Language Implementations with Alastair Donaldson</li> + <li>Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might</li> + <li>The evolution of Scala with Martin Odersky</li></ul> + +<p>A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.</p> + +<p>Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.</p> + +<p>More information at:</p> + +<p><a href="https://pliss2017.github.io">https://pliss2017.github.io</a></p> + +<p>(Oregon is a reference to the <a href="https://www.cs.uoregon.edu/research/summerschool/summer17/">OPLSS</a>, in which you may also be interested.)</p> \ No newline at end of file diff --git a/blog/feeds/language.atom.xml b/blog/feeds/language.atom.xml new file mode 100644 index 00000000..08fe8435 --- /dev/null +++ b/blog/feeds/language.atom.xml @@ -0,0 +1,733 @@ + + + PRL Blog: Posts tagged 'language' + + + urn:http-prl-ccs-neu-edu:-blog-tags-language-html + 2018-10-22T15:05:17Z + + Defining Local Bindings in Turnstile Languages + + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + 2018-10-22T15:05:17Z + 2018-10-22T15:05:17Z + + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> \ No newline at end of file diff --git a/blog/feeds/language.rss.xml b/blog/feeds/language.rss.xml new file mode 100644 index 00000000..a9fc1f61 --- /dev/null +++ b/blog/feeds/language.rss.xml @@ -0,0 +1,733 @@ + + + + PRL Blog: Posts tagged 'language' + PRL Blog: Posts tagged 'language' + http://prl.ccs.neu.edu/blog/tags/language.html + Mon, 22 Oct 2018 15:05:17 UT + Mon, 22 Oct 2018 15:05:17 UT + 1800 + + Defining Local Bindings in Turnstile Languages + http://prl.ccs.neu.edu/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/?utm_source=language&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + Mon, 22 Oct 2018 15:05:17 UT + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> \ No newline at end of file diff --git a/blog/feeds/latency.atom.xml b/blog/feeds/latency.atom.xml new file mode 100644 index 00000000..50563c9e --- /dev/null +++ b/blog/feeds/latency.atom.xml @@ -0,0 +1,456 @@ + + + PRL Blog: Posts tagged 'latency' + + + urn:http-prl-ccs-neu-edu:-blog-tags-latency-html + 2016-05-24T10:51:34Z + + Measuring GC latencies in Haskell, OCaml, Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + 2016-05-24T10:51:34Z + 2016-05-24T10:51:34Z + + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/latency.rss.xml b/blog/feeds/latency.rss.xml new file mode 100644 index 00000000..e389ecfc --- /dev/null +++ b/blog/feeds/latency.rss.xml @@ -0,0 +1,456 @@ + + + + PRL Blog: Posts tagged 'latency' + PRL Blog: Posts tagged 'latency' + http://prl.ccs.neu.edu/blog/tags/latency.html + Tue, 24 May 2016 10:51:34 UT + Tue, 24 May 2016 10:51:34 UT + 1800 + + Measuring GC latencies in Haskell, OCaml, Racket + http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/?utm_source=latency&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + Tue, 24 May 2016 10:51:34 UT + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/lectures.atom.xml b/blog/feeds/lectures.atom.xml new file mode 100644 index 00000000..3db84d3b --- /dev/null +++ b/blog/feeds/lectures.atom.xml @@ -0,0 +1,83 @@ + + + PRL Blog: Posts tagged 'lectures' + + + urn:http-prl-ccs-neu-edu:-blog-tags-lectures-html + 2019-03-09T14:40:16Z + + PLISS: Learn About PL Implementation in a Castle + + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + 2019-03-09T14:40:16Z + 2019-03-09T14:40:16Z + + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> + + PLISS: Oregon without the greek + + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-pliss-oregon-without-the-greek + 2017-02-28T23:01:00Z + 2017-02-28T23:01:00Z + + Jan Vitek + +<p>What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or <a href="https://pliss2017.github.io">PLISS</a> for short) has set out to answer.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2017.github.io/speakers.html">twelve speakers</a> versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.</p> + +<p>Lectures cover current research and future trends in programming language design and implementation, including:</p> + +<ul> + <li>Writing Just-in-time Compilers with LLVM with Jan Vitek</li> + <li>Performance Evaluation and Benchmarking with Laurie Tratt</li> + <li>Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller</li> + <li>High-Performance Fully Concurrent Garbage Collection with Richard Jones</li> + <li>Compiling Dynamic Languages with David Edelsohn</li> + <li>Language-support for Distributed Datastores with Suresh Jagannathan</li> + <li>Testing Programming Language Implementations with Alastair Donaldson</li> + <li>Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might</li> + <li>The evolution of Scala with Martin Odersky</li></ul> + +<p>A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.</p> + +<p>Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.</p> + +<p>More information at:</p> + +<p><a href="https://pliss2017.github.io">https://pliss2017.github.io</a></p> + +<p>(Oregon is a reference to the <a href="https://www.cs.uoregon.edu/research/summerschool/summer17/">OPLSS</a>, in which you may also be interested.)</p> \ No newline at end of file diff --git a/blog/feeds/lectures.rss.xml b/blog/feeds/lectures.rss.xml new file mode 100644 index 00000000..698fb323 --- /dev/null +++ b/blog/feeds/lectures.rss.xml @@ -0,0 +1,81 @@ + + + + PRL Blog: Posts tagged 'lectures' + PRL Blog: Posts tagged 'lectures' + http://prl.ccs.neu.edu/blog/tags/lectures.html + Sat, 09 Mar 2019 14:40:16 UT + Sat, 09 Mar 2019 14:40:16 UT + 1800 + + PLISS: Learn About PL Implementation in a Castle + http://prl.ccs.neu.edu/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/?utm_source=lectures&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-03-09-pliss-learn-about-pl-implementation-in-a-castle + Sat, 09 Mar 2019 14:40:16 UT + Alexi Turcotte + +<p>We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you&rsquo;re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (<a href="https://pliss2019.github.io/">PLISS</a> for short).</p> + +<p>PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is <em>March 29th, 2019</em> at <em>17:00 GMT</em>. More details can be found <a href="https://pliss2019.github.io/registration.html">here</a>.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2019.github.io/speakers.html">ten speakers</a> from both academia and industry, each well-versed in the practical side of programming languages. The lectures cover current research as well as future trends in programming language design and implementation, including:</p> + +<ul> + <li>Developing Security-Aware Languages with Cristina Cifuentes;</li> + <li>Semantics-First Language Design with Sylvan Clebsch;</li> + <li>Compiler Design Patterns for Machine Learning by Albert Cohen;</li> + <li>Design and Analysis of Configuration Languages by Arjun Guha;</li> + <li>A Survey of V8 and WebAssembly by Ben L. Titzer;</li> + <li>Crafting User-Friendly Compilers by Nicholas Matsakis;</li> + <li>Static Program Analysis by Anders Møller;</li> + <li>How Industry Approaches Language and Compiler Design by Joe Pamer;</li> + <li>What an End to Non-Volatile RAM Means for Researchers by Mario Wolczko.</li></ul> + +<p>Besides attending lectures, students will also be able to get to know the speakers and attendees and think of new research problems. A week-long stay in beautiful Bertinoro, Italy is an ideal setting for socializing with other PL enthusiasts and building lasting relationships.</p> + +<p>If I may, I attended the first PLISS in 2017 and can&rsquo;t recommend it enough. The atmosphere at summer schools is truly unparalleled, and I made friends there that have stood the test of time. For what it&rsquo;s worth to any prospective graduate students, PLISS is also where I met my PhD advisor. Students will be surprised at how many faces they recognize at future conferences, and in a sense summer schools are nice introduction to the research community. You can read another attendee&rsquo;s testimonial <a href="http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/">here</a>.</p> + +<p>More information can be found at:</p> + +<p><a href="https://pliss2019.github.io">https://pliss2019.github.io</a></p> + +<p>(No, really, it&rsquo;s in a castle. Look at the pictures.)</p> + + PLISS: Oregon without the greek + http://prl.ccs.neu.edu/blog/2017/02/28/pliss-oregon-without-the-greek/?utm_source=lectures&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-02-28-pliss-oregon-without-the-greek + Tue, 28 Feb 2017 23:01:00 UT + Jan Vitek + +<p>What does every student interested in programming languages need to learn about the practical side of the field? That is the question that the first international summer school on programming language implementation (or <a href="https://pliss2017.github.io">PLISS</a> for short) has set out to answer.</p> +<!-- more--> + +<p><img src="/img/pliss_summer_school_2017_logo.png" alt="PLISS logo" /></p> + +<p>The school will feature <a href="https://pliss2017.github.io/speakers.html">twelve speakers</a> versed in programming language practicae ranging from abstract interpretation to garbage collection and from compiler implementation to language design. The lectures will feature hands on exercises as well as lessons learned from large scale industrial efforts.</p> + +<p>Lectures cover current research and future trends in programming language design and implementation, including:</p> + +<ul> + <li>Writing Just-in-time Compilers with LLVM with Jan Vitek</li> + <li>Performance Evaluation and Benchmarking with Laurie Tratt</li> + <li>Designing a Commercial Actor Language with Sophia Drossopoulou and Heather Miller</li> + <li>High-Performance Fully Concurrent Garbage Collection with Richard Jones</li> + <li>Compiling Dynamic Languages with David Edelsohn</li> + <li>Language-support for Distributed Datastores with Suresh Jagannathan</li> + <li>Testing Programming Language Implementations with Alastair Donaldson</li> + <li>Abstract Interpretation and Static Analysis with lectures by Francesco Logozzo and Matt Might</li> + <li>The evolution of Scala with Martin Odersky</li></ul> + +<p>A summer school is also an opportunity to get know the speakers and get ideas for research problems. Students will have the opportunity to socialize with a peer group of other students in PL as well as professors and industrial researchers.</p> + +<p>Thanks to generous funding from NSF, ONR and SIGPLAN, costs will be kept low and some fellowships are available to cover travel costs.</p> + +<p>More information at:</p> + +<p><a href="https://pliss2017.github.io">https://pliss2017.github.io</a></p> + +<p>(Oregon is a reference to the <a href="https://www.cs.uoregon.edu/research/summerschool/summer17/">OPLSS</a>, in which you may also be interested.)</p> \ No newline at end of file diff --git a/blog/feeds/lost-time.atom.xml b/blog/feeds/lost-time.atom.xml new file mode 100644 index 00000000..fc73e44f --- /dev/null +++ b/blog/feeds/lost-time.atom.xml @@ -0,0 +1,62 @@ + + + PRL Blog: Posts tagged 'lost time' + + + urn:http-prl-ccs-neu-edu:-blog-tags-lost-time-html + 2016-08-03T14:09:02Z + + A few cores too many + + urn:http-prl-ccs-neu-edu:-blog-2016-08-03-a-few-cores-too-many + 2016-08-03T14:09:02Z + 2016-08-03T14:09:02Z + + Ben Greenman + +<p>Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.</p> +<!-- more--> + +<p>Last year, we proposed a method for evaluating the performance of gradual type systems based on measuring <em>every possible configuration</em> of typed and untyped code that a programmer might explore <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">(pdf)</a>. Given the freedom that gradual typing offers, this is the only realistic way to measure the performance of a gradual type system.</p> + +<p>But it is a lot to measure! While developing the method, we spent over 3 months benchmarking a total of 75,844 configurations. Each configuration is a complete program and some gradual typings caused some programs to slow by 50x or even 100x, so many of these configurations took minutes to run.</p> + +<p>The next question we asked was naturally &ldquo;how can we scale this method to large software projects?&rdquo; In <a href="http://docs.racket-lang.org/ts-reference/Libraries_Provided_With_Typed_Racket.html#%28part._.Porting_.Untyped_.Modules_to_.Typed_.Racket%29">our case</a>, the number of gradually typed configurations scaled exponentially with the number of modules. Current gradual type system for <a href="https://github.com/mvitousek/reticulated">Python</a> and <a href="http://www.di.ens.fr/~zappa/readings/ecoop15.pdf">JavaScript</a> are exponential in the number of <em>variables</em> in the program.</p> + +<p>We explored two solutions:</p> + +<ol> + <li>Max New began work on a prediction model (inspired by work on <a href="http://subs.emis.de/LNI/Proceedings/Proceedings213/185.pdf">software product lines</a>) to estimate the performance of <code>2^N</code> configurations after polynomially-many measurements.</li> + <li>Asumu Takikawa and I shopped for a multi-core computer.</li></ol> + +<p>By Thanksgiving, we had bought a Linux machine with 2 <a href="http://www.amd.com/en-us/products/server/opteron/6000/6300">AMD Opteron 6376 2.3GHz</a> processors (16 cores each) and put it to work running benchmarks on 29 cores simultaneously. Life was good.</p> + +<p>Later that winter, Max implemented a prediction algorithm. The basic idea was to focus on <em>boundaries</em> between modules and isolate their effect on performance. If two modules are untyped, their boundary will have zero cost. If the same two modules are typed, their boundary might result in an overall performance improvement due to type-driven optimizations. And if one module is typed and the other untyped, their boundary will suffer some cost of type checking at runtime. In general a program with <code>N</code> modules has at most <code>N(N - 1) / 2</code> internal boundaries, so it is far more time-efficient to measure only the boundaries than to benchmark <code>2^N</code> gradually typed configurations.</p> + +<p>Fast-forward to March, we had a prototype prediction algorithm and it was time to test. Again using 29 cores (because, why not), we gathered cost/benefit numbers for one 4-module benchmark and used them to predict performance for its 16 configurations. The results were not very good.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-1.png" alt="Figure 1: True running time vs. predicted running time for 16 configurations" /> + <p class="caption">Figure 1: True running time vs. predicted running time for 16 configurations</p></div> + +<p>Those green circles are the ground truth, the average running time after 5 iterations of each config. The blue triangles are what we predicted. Except for configurations 0 and 8, the triangles are FAR off from the truth. Many are even negative &hellip; obviously the algorithm needs work.</p> + +<p>But then, out of frustration, desperation, or just good luck, Max compared the predictions to ground truth data gathered on a <em>single</em> core, leaving the other 31 cores idle.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-2.png" alt="Figure 2: Predictions made using measurements from a single core" /> + <p class="caption">Figure 2: Predictions made using measurements from a single core</p></div> + +<p>First off, the red &ldquo;sequential truth&rdquo; dots are slightly closer to the predicted triangles. Second &mdash; and this is the scary part &mdash; the red dots are very different from the green dots. <em>Running on 1 core vs. 29 cores should not change the measurements!</em></p> + +<p>From here we tried increasing the running time of the benchmark, removing I/O and system calls, checking for hyperthreading (ARM cores don&rsquo;t support it), and even changing the cores&rsquo; CPU governor. The hope was that results taken from 1 core could match results from <code>N</code> cores, for some <code>N &gt; 1</code>. It turns out <code>N = 2</code> was stable, but even for <code>N = 3</code> we found graphs like the following:</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-3.png" alt="Figure 3: exact running times. Same-colored dots in each column should be tightly clustered." /> + <p class="caption">Figure 3: exact running times. Same-colored dots in each column should be tightly clustered.</p></div> + +<p>This data is for the same 16 configurations as the previous two graphs. Green dots are exact running times measured with 25 cores. Red dots are running times measured with 1 core. The red dots are much closer together, and always unimodal. The green dots are evidence that maybe the 32-core machine has, as Jan Vitek put it, 30 cores too many.</p> + +<blockquote> + <p>&ldquo;Oh my. You think it&rsquo;ll never happen to you. Well, now I&rsquo;ve learned my lesson.&rdquo;</p><!-- bg: If anyone knows this quote I will be AMAZED. If anyone can even Google this quote, I'll buy them 2 beers and a pizza.--></blockquote> + +<p>And so, we said goodbye to the last 4 months of data and started over running at most two cores. The new results are all stable, but still we keep pinching ourselves.</p> + +<p>P.S. the results from <a href="http://www.ccs.neu.edu/racket/pubs/#popl16-tfgnvf">POPL 2016</a> are just fine, as they were not taken on the new machine running more than 2 cores. If you have time to confirm, that data is in our <a href="http://www.ccs.neu.edu/home/asumu/artifacts/popl-2016/">artifact</a> and in the <a href="https://github.com/nuprl/gradual-typing-performance/tree/master/paper/popl-2016/data">gradual-typing-performance</a> repo.</p> \ No newline at end of file diff --git a/blog/feeds/lost-time.rss.xml b/blog/feeds/lost-time.rss.xml new file mode 100644 index 00000000..c57ca2e1 --- /dev/null +++ b/blog/feeds/lost-time.rss.xml @@ -0,0 +1,62 @@ + + + + PRL Blog: Posts tagged 'lost time' + PRL Blog: Posts tagged 'lost time' + http://prl.ccs.neu.edu/blog/tags/lost-time.html + Wed, 03 Aug 2016 14:09:02 UT + Wed, 03 Aug 2016 14:09:02 UT + 1800 + + A few cores too many + http://prl.ccs.neu.edu/blog/2016/08/03/a-few-cores-too-many/?utm_source=lost-time&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-08-03-a-few-cores-too-many + Wed, 03 Aug 2016 14:09:02 UT + Ben Greenman + +<p>Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.</p> +<!-- more--> + +<p>Last year, we proposed a method for evaluating the performance of gradual type systems based on measuring <em>every possible configuration</em> of typed and untyped code that a programmer might explore <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">(pdf)</a>. Given the freedom that gradual typing offers, this is the only realistic way to measure the performance of a gradual type system.</p> + +<p>But it is a lot to measure! While developing the method, we spent over 3 months benchmarking a total of 75,844 configurations. Each configuration is a complete program and some gradual typings caused some programs to slow by 50x or even 100x, so many of these configurations took minutes to run.</p> + +<p>The next question we asked was naturally &ldquo;how can we scale this method to large software projects?&rdquo; In <a href="http://docs.racket-lang.org/ts-reference/Libraries_Provided_With_Typed_Racket.html#%28part._.Porting_.Untyped_.Modules_to_.Typed_.Racket%29">our case</a>, the number of gradually typed configurations scaled exponentially with the number of modules. Current gradual type system for <a href="https://github.com/mvitousek/reticulated">Python</a> and <a href="http://www.di.ens.fr/~zappa/readings/ecoop15.pdf">JavaScript</a> are exponential in the number of <em>variables</em> in the program.</p> + +<p>We explored two solutions:</p> + +<ol> + <li>Max New began work on a prediction model (inspired by work on <a href="http://subs.emis.de/LNI/Proceedings/Proceedings213/185.pdf">software product lines</a>) to estimate the performance of <code>2^N</code> configurations after polynomially-many measurements.</li> + <li>Asumu Takikawa and I shopped for a multi-core computer.</li></ol> + +<p>By Thanksgiving, we had bought a Linux machine with 2 <a href="http://www.amd.com/en-us/products/server/opteron/6000/6300">AMD Opteron 6376 2.3GHz</a> processors (16 cores each) and put it to work running benchmarks on 29 cores simultaneously. Life was good.</p> + +<p>Later that winter, Max implemented a prediction algorithm. The basic idea was to focus on <em>boundaries</em> between modules and isolate their effect on performance. If two modules are untyped, their boundary will have zero cost. If the same two modules are typed, their boundary might result in an overall performance improvement due to type-driven optimizations. And if one module is typed and the other untyped, their boundary will suffer some cost of type checking at runtime. In general a program with <code>N</code> modules has at most <code>N(N - 1) / 2</code> internal boundaries, so it is far more time-efficient to measure only the boundaries than to benchmark <code>2^N</code> gradually typed configurations.</p> + +<p>Fast-forward to March, we had a prototype prediction algorithm and it was time to test. Again using 29 cores (because, why not), we gathered cost/benefit numbers for one 4-module benchmark and used them to predict performance for its 16 configurations. The results were not very good.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-1.png" alt="Figure 1: True running time vs. predicted running time for 16 configurations" /> + <p class="caption">Figure 1: True running time vs. predicted running time for 16 configurations</p></div> + +<p>Those green circles are the ground truth, the average running time after 5 iterations of each config. The blue triangles are what we predicted. Except for configurations 0 and 8, the triangles are FAR off from the truth. Many are even negative &hellip; obviously the algorithm needs work.</p> + +<p>But then, out of frustration, desperation, or just good luck, Max compared the predictions to ground truth data gathered on a <em>single</em> core, leaving the other 31 cores idle.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-2.png" alt="Figure 2: Predictions made using measurements from a single core" /> + <p class="caption">Figure 2: Predictions made using measurements from a single core</p></div> + +<p>First off, the red &ldquo;sequential truth&rdquo; dots are slightly closer to the predicted triangles. Second &mdash; and this is the scary part &mdash; the red dots are very different from the green dots. <em>Running on 1 core vs. 29 cores should not change the measurements!</em></p> + +<p>From here we tried increasing the running time of the benchmark, removing I/O and system calls, checking for hyperthreading (ARM cores don&rsquo;t support it), and even changing the cores&rsquo; CPU governor. The hope was that results taken from 1 core could match results from <code>N</code> cores, for some <code>N &gt; 1</code>. It turns out <code>N = 2</code> was stable, but even for <code>N = 3</code> we found graphs like the following:</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-3.png" alt="Figure 3: exact running times. Same-colored dots in each column should be tightly clustered." /> + <p class="caption">Figure 3: exact running times. Same-colored dots in each column should be tightly clustered.</p></div> + +<p>This data is for the same 16 configurations as the previous two graphs. Green dots are exact running times measured with 25 cores. Red dots are running times measured with 1 core. The red dots are much closer together, and always unimodal. The green dots are evidence that maybe the 32-core machine has, as Jan Vitek put it, 30 cores too many.</p> + +<blockquote> + <p>&ldquo;Oh my. You think it&rsquo;ll never happen to you. Well, now I&rsquo;ve learned my lesson.&rdquo;</p><!-- bg: If anyone knows this quote I will be AMAZED. If anyone can even Google this quote, I'll buy them 2 beers and a pizza.--></blockquote> + +<p>And so, we said goodbye to the last 4 months of data and started over running at most two cores. The new results are all stable, but still we keep pinching ourselves.</p> + +<p>P.S. the results from <a href="http://www.ccs.neu.edu/racket/pubs/#popl16-tfgnvf">POPL 2016</a> are just fine, as they were not taken on the new machine running more than 2 cores. If you have time to confirm, that data is in our <a href="http://www.ccs.neu.edu/home/asumu/artifacts/popl-2016/">artifact</a> and in the <a href="https://github.com/nuprl/gradual-typing-performance/tree/master/paper/popl-2016/data">gradual-typing-performance</a> repo.</p> \ No newline at end of file diff --git a/blog/feeds/math.atom.xml b/blog/feeds/math.atom.xml new file mode 100644 index 00000000..6344207b --- /dev/null +++ b/blog/feeds/math.atom.xml @@ -0,0 +1,520 @@ + + + PRL Blog: Posts tagged 'math' + + + urn:http-prl-ccs-neu-edu:-blog-tags-math-html + 2017-09-27T15:44:57Z + + Final Algebra Semantics is Observational Equivalence + + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + 2017-09-27T15:44:57Z + 2017-09-27T15:44:57Z + + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> + + Closure Conversion as CoYoneda + + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + 2017-08-28T10:30:00Z + 2017-08-28T10:30:00Z + + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> + + Understanding Constructive Galois Connections + + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + 2016-11-16T00:00:00Z + 2016-11-16T00:00:00Z + + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/math.rss.xml b/blog/feeds/math.rss.xml new file mode 100644 index 00000000..aebf92c5 --- /dev/null +++ b/blog/feeds/math.rss.xml @@ -0,0 +1,516 @@ + + + + PRL Blog: Posts tagged 'math' + PRL Blog: Posts tagged 'math' + http://prl.ccs.neu.edu/blog/tags/math.html + Wed, 27 Sep 2017 15:44:57 UT + Wed, 27 Sep 2017 15:44:57 UT + 1800 + + Final Algebra Semantics is Observational Equivalence + http://prl.ccs.neu.edu/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/?utm_source=math&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + Wed, 27 Sep 2017 15:44:57 UT + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> + + Closure Conversion as CoYoneda + http://prl.ccs.neu.edu/blog/2017/08/28/closure-conversion-as-coyoneda/?utm_source=math&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-08-28-closure-conversion-as-coyoneda + Mon, 28 Aug 2017 10:30:00 UT + Max New + +<p>The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, <em>typed</em> versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (<a href="http://www.ccs.neu.edu/home/amal/papers/epc.pdf">cps</a>,<a href="http://www.ccs.neu.edu/home/amal/papers/tccpoe.pdf">cc</a>).</p> + +<p>In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the <a href="https://ncatlab.org/nlab/show/co-Yoneda+lemma">*co*Yoneda lemma</a>.</p> + +<p>The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the <a href="http://www.cs.ox.ac.uk/people/daniel.james/iso/iso.pdf">literature</a> and blogosphere (<a href="https://golem.ph.utexas.edu/category/2008/01/the_continuation_passing_trans.html">ncafe</a>, <a href="https://bartoszmilewski.com/2015/09/01/the-Yoneda-lemma/">Bartosz</a>), so I&rsquo;ll focus on closure conversion here. Also, I&rsquo;ll try to go into some detail in showing how the &ldquo;usual&rdquo; version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.</p> + +<p>I&rsquo;ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a <a href="http://matt.might.net/articles/closure-conversion/">nice blog post explaining untyped closure conversion</a>.</p> +<!-- more--> + +<p>\( +\newcommand{\Set}{\mathsf{Set}} +\newcommand{\Hom}{\mathsf{Hom}} +\)</p> + +<h2 id="polymorphic-closure-conversion">Polymorphic Closure Conversion</h2> + +<p>Closure conversion is a way of compiling a language with closures (i.e., basically any modern high-level language) to one that only has function pointers/labels like C or machine code. Closure conversion compiles high-level functions (aka closures) to a pair of an environment that will contain the values of all the functions&rsquo; free variables and a code pointer to a block that takes as inputs all the inputs to the function and values for all of the free variables.</p> + +<p>For instance</p> + +<pre><code>let x = 3 in λ y. x + y</code></pre> + +<p>would be converted to something like</p> + +<pre><code>let x = 3 in ([x: 3], λ env, y. let x = env.x in x + y)</code></pre> + +<p>Can we give a type to the resulting code? The source program has type <code>Number -&gt; Number</code>, but the target has a type more like</p> + +<pre><code>{ x: Number} × ({x : Number} × Number -&gt; Number).</code></pre> + +<p>In addition to being ugly, this type is leaking irrelevant details of the function&rsquo;s implementation: all of its free variables are there in its type, so two terms with the same function type but different free variables would be translated to different types. Also high-level program equivalences like \(\beta\)-reducing the term to just <code>λ y. 3 + y</code> would not even preserve typing. Not only that, but some bad code could now supply a <em>different</em>, well-typed value for <code>x</code> than allowed which could break invariants the programmer had about the function.</p> + +<p>We could fix the type preservation issue by just using a dynamic type for our environment, but this would still leak details in the values. Fortunately, there is a nice solution to the other problems using existential types. The idea is that the type of the environment of free variables is <em>irrelevant</em> to anyone that calls the function, only the function itself should know what the environment looks like; the type of the environment should be <em>abstract</em> to the caller and <em>concrete</em> to the callee. Existential types capture this.</p> + +<p>We can translate functions in the source of type <code>A -&gt; B</code> to pairs of an environment and a code pointer, but now making the environment type existentially quantified:</p> + +<pre><code>∃ Γ. Γ × (Γ × A -&gt; B).</code></pre> + +<p>Then the syntax of existential types ensure that all any consumer can do with the <code>env : Γ</code> in the pair is pass it to the code pointer with an <code>A</code> argument.</p> + +<p>How do we prove that this is correct? And what does correct even mean? We&rsquo;ll focus on a property called <em>full abstraction</em> which says that if two programs are equal in the source language, then their translations are equal. Here, equal in the source language will just mean \(\beta,\eta \) equivalence, so things like as above:</p> + +<pre><code>let x = 3 in λ y. x + y +≡ +λ y. 3 + y</code></pre> + +<p>To prove this we&rsquo;ll show that in a language with existential types the types <code>∃ Γ. Γ × (Γ × A -&gt; B)</code> and <code>A \to B</code> are isomorphic. The usual proof is by parametricity, instead we&rsquo;ll use a closely related category-theoretic argument: the coYoneda lemma.</p> + +<h2 id="the-coyoneda-lemma">The CoYoneda Lemma</h2> + +<p>The coYoneda lemma is a generalization of the equivalence described above. I&rsquo;ll start with the ordinary version which uses <em>coends</em> and <em>presheaves</em>.</p> + +<p>The coYoneda lemma says that for any category \( C \), presheaf \( Q : C^{op} \to \Set \), and object \(A \in C \), \(Q(A) \) is isomorphic to the coend: \[ \exists B. (A \to B) \times Q(B) \] Let&rsquo;s break that down.</p> + +<h3 id="coends">Coends</h3> + +<p>A coend is a construction that is very similar to the parametric existential quantifier. If you&rsquo;re familiar with parametricity, a good intuition is that coends have the same definition as existential types but where the only relations are functional relations.</p> + +<p>You can take the coend of a functor of type \(M : C^{op} \times C \to +\Set \). We can get such an \(M \) from a type with a free type variable like \( X \times A \to X \) by splitting the \(X \) into positive and negative occurrences: \(X^- \times A \to X^+ \). Then the coend \(\exists X. M(X,X) \in \Set \) is like the union of all \(M(X,X) \), but where the \(X \) is ensured to be &ldquo;irrelevant&rdquo;.</p> + +<p>So for any object \(A \in C \) there is a map \(pack_A : M(A,A) \to +\exists X. M(X,X) \), we can &ldquo;hide the A&rdquo;. To make sure the \(X \) is treated opaquely, we add an invariance condition that says if you have an \(mA : M(A,A) \) and an \(mB : +M(B,B) \) such that the \(A, B\) positions are related by some function \(f : A \to B \), then \(pack_A(mA) = pack_B(mB)\). More formally, this means that if you have a \(m' : M(B,A) \), then</p> + +<p>\[ pack_B(M(B,f)(m')) = pack_A(M(f,A)(m'))\] or in a point-free style: \[ pack_B \circ M(B,f) = pack_A \circ M(f,A) : M(B,A) \to \exists X. M(X,X) \]</p> + +<p>A function parameterized by types like \(pack \) that has this property is called a <em>co-wedge</em> from \(M \).</p> + +<p>A coend is an object \(\exists X. M(X,X) \) and a co-wedge \(\forall +A. pack_A : M(A,A) \to \exists X. M(X,X) \) that are <em>universal</em>, i.e. any other co-wedge \(\forall A. f_A : M(A,A) \to C\) factors through \(pack_A \). This gives us the syntax for existential elimination.</p> + +<p>If you are familiar with parametricity, it is a good exercise to see why the usual condition for invariance wrt all <em>relations</em> implies that a parametric \(pack, \exists X. M(X,X) \) will form a cowedge. It seems that in general it would not be a universal co-wedge because a parametric exists is invariant under all relations and there are many relations that don&rsquo;t act like functions.</p> + +<h3 id="presheaves">Presheaves</h3> + +<p>Next, a presheaf is just a functor \( Q : C^{op} \to +\Set \). Think of this as a set that is parameterised by a type of &ldquo;inputs&rdquo;, so if you have a map in \(C, f : A \to B\) you get a function \(Q(f) : +Q(B) \to Q(A) \) that &ldquo;preprocesses&rdquo; the inputs using \(f +\). Functoriality ensures that preprocessing with the identity is just the identity and that composition of preprocessers is the preprocessor from the composite function.</p> + +<p>So the informal explanation of the coYoneda lemma is that for any presheaf \(Q \), if we have an \( \exists X. (A \to X) \times Q(X) +\), then since we can&rsquo;t inspect the \(X \) in any way, all we can really do is compose the \(Q(X) \) with the preprocesser from the function \(A \to X \), giving us a \(Q(A) \).</p> + +<h3 id="enriched-categories-and-enriched-coyoneda">Enriched Categories and Enriched CoYoneda</h3> + +<p>But there&rsquo;s a gap from here to applying this to a programming language, the coYoneda lemma as presented says that \(Q(A) \) and \(\exists B. (A \to B) \times Q(B) \) are isomorphic as <em>sets</em>, but we wanted an isomorphism of <em>types</em> in our programming language. We can reconcile this by considering <em>enriched</em> category theory and the <em>enriched</em> coYoneda lemma. Let \(V \) be a category, then if \( V +\) is sufficiently like the category of sets, then we can do a lot of category theory by replacing the word &ldquo;set&rdquo; with &ldquo;object of \(V \)&rdquo;.</p> + +<p>Specifically, a \(V \)-enriched category (or just \(V\)-category) has a set of objects \(Ob \), but for each pair of objects \(A,B +\in Ob \) we get a \( V\)-object \(\Hom(A,B) \) of morphisms from \(A \) to \( B \). If \(V \) is a closed category, we can see \(V \) <em>itself</em> as a \(V\)-enriched category with the same objects and just making \(\Hom(A,B) = A \to B \) i.e. the <em>internal</em> hom aka exponential.</p> + +<p>Then we can reinterpret the coYoneda lemma above by saying \(C \) is a \(V\)-category and \(Q \) is a \(V\)-presheaf i.e., just a contravariant functor from \(V \) to itself: \(Q : V^{op} \to V\) where the preprocessing function is now a morphism in \(C \). Haskelletons just call this a <a href="https://hackage.haskell.org/package/contravariant-1.4/docs/Data-Functor-Contravariant.html">contravariant functor</a>. Furthermore, since existential types provide at least as strong of a reasoning principle as coends, the proof of the coYoneda lemma goes through with existential types instead. Finally, the point-free description above for coend can be interpreted in any category.</p> + +<p>Now that we&rsquo;re working all inside our language, let&rsquo;s look at what the isomorphism looks like in Haskellish/Agdaish syntax. We want mutually inverse functions</p> + +<pre><code>f : (Contravariant Q) =&gt; (∃ Γ. (Δ -&gt; Γ) × (Q Γ)) -&gt; Q Δ +g : (Contravariant Q) =&gt; Q Δ -&gt; ∃ Γ. (Δ -&gt; Γ) × (Q Γ)</code></pre> + +<p>If you try to implement them you won&rsquo;t be able to get it wrong, but here they are:</p> + +<pre><code>f (k, qΓ) = contramap k qΓ +g qΔ = (id, qΔ)</code></pre> + +<p>where we just instantiate \(\Gamma = \Delta \) in the second case. You can prove \( f \circ g = id \) using just \(\beta \) and the Contravariant laws, but to prove \(g \circ f = id \) you need to use the coend reasoning. For those of you that know about the Yoneda lemma, note the similarity to that proof in using the identity function and instantiating a type variable in a trivial way.</p> + +<h2 id="closure-version-as-coyoneda">Closure Version as CoYoneda</h2> + +<p>Now it&rsquo;s time to bring it all together. Let \(V \) be our programming language viewed as a category in the usual way.</p> + +<p>We want to prove the closure conversion isomorphism:</p> + +<p>\[ A \to B \cong \exists \Gamma. \Gamma \times (\Gamma \times A \to B) +\]</p> + +<p>using the \(V \)-coYoneda lemma which says for any contravariant functor \(Q : V^{op} \to V \), and object \(\Delta \in V\)</p> + +<p>\[ Q(\Delta) \cong \exists \Gamma. (\Delta \to \Gamma) \times Q(\Gamma) +\]</p> + +<p>Clearly based on the right hand side, \(Q \) should be \( - \times +A \to B \) which gives us for any \(\Delta \in V\):</p> + +<p>\[ \Delta \times A \to B \cong \exists \Gamma. (\Delta \to \Gamma) \times (\Gamma \times A \to B) +\]</p> + +<p>Next we pick \(\Delta = 1\), the unit type. Then we use some basic facts about the unit type: \(1 \times A \cong +A \) and \(1 \to \Gamma \cong \Gamma\) (at least in a pure language) to get the desired result by composition:</p> + +<p>\[ A \to B \cong 1 \times A \to B \cong \exists \Gamma. (1 \to +\Gamma) \times (\Gamma \times A \to B) \cong \exists \Gamma. \Gamma +\times (\Gamma \times A \to B)\]</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Since closure conversion is an instance of the CoYoneda lemma, this might be a nice example to give intuition for CoYoneda for programmers. While not as famous as its cousin Yoneda, CoYoneda is used in <a href="https://hackage.haskell.org/package/kan-extensions-5.0.2/docs/Data-Functor-Coyoneda.html">Haskell</a> and is also central to the <a href="https://ncatlab.org/nlab/show/Day+convolution">Day Convolution</a>, which can be used to give semantics to <a href="atkey-thesis">separation logic</a>.</p> + +<p>Also in researching for this post, I was surprised at how little I could find on the relationship between ends/coends and relational parametricity. This seems very unfortunate as it looks like we&rsquo;re reproving some of the same theorems (Yoneda, coYoneda) using very similar, but incompatible formalisms.</p> + +<h2 id="you-might-also-like">You might also like</h2> + +<ul> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/">Syntactic Parametricity Strikes Again</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/">Categorical Semantics for Dynamically Typed Programming Languages</a></p></li> + <li> + <p><a href="http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/">Understanding Constructive Galois Connections</a>.</p></li></ul> + + Understanding Constructive Galois Connections + http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/?utm_source=math&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-16-understanding-constructive-galois-connections + Wed, 16 Nov 2016 00:00:00 UT + Max New + +<p>One of my favorite papers at ICFP 2016 (in lovely <a href="http://conf.researchr.org/home/icfp-2016">Nara, Japan</a>) was <a href="https://arxiv.org/abs/1511.06965">Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory</a> by <a href="http://david.darais.com/">David Darais</a> and <a href="https://www.cs.umd.edu/~dvanhorn/">David Van Horn</a>. The central technical result is quite interesting, but a little intimidating, so I&rsquo;d like to share a &ldquo;de-generalization&rdquo; of the result that I found helpful to understand.</p> +<!-- more--> + +<h1 id="history">History</h1> + +<p>I won&rsquo;t go into much of the details of the paper, because I think it is quite well written, but here&rsquo;s a short overview. The paper is about how to do verified static analysis while taking advantage of the calculational approach of <a href="http://www.di.ens.fr/~cousot/COUSOTpapers/Marktoberdorf98.shtml">Abstract Interpretation</a>. The problem is that the Galois connections people use for abstract domains are not always computable. Darais and Van Horn show however that there is a very useful class of Galois connections that is computable, and they show how they can exploit this to write verified static analyses that more closely follow the &ldquo;on-paper&rdquo; proofs, and offload much of the details to the proof assistant as mere calculation.</p> + +<p>David Darais told me about these results when we were at POPL 2016 (in less lovely but much more convenient <a href="http://conf.researchr.org/home/POPL-2016">St. Petersburg, Florida</a>) and in particular about the central theorem of the paper, which shows that two different classes of Galois connections they define, &ldquo;Kleisli&rdquo; and &ldquo;Constructive&rdquo; Galois connections, are actually constructively equivalent. I was really surprised by the result when he explained it to me, and so I hoped to find if there was a known generalization of the result for adjunctions of categories, rather than Galois connections of posets.</p> + +<p>Eventually, my usual trawling of <a href="http://mathoverflow.net/">Mathoverflow</a> and <a href="https://ncatlab.org/nlab/show/HomePage">nlab</a> led me to a <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">not-quite generalization to categories</a> and interestingly a <a href="http://mathoverflow.net/questions/222516/duality-between-compactness-and-hausdorffness/222524#222524"><em>de</em>-generalization to sets</a> that helped me immensely to understand the theorem.</p> + +<p>Since I know that the original theorem is a bit technical, I&rsquo;ll explain the de-generalization to sets here, which I hope will help to understand their theorem.</p> + +<h1 id="functions-and-relations">Functions and Relations</h1> + +<p>Let&rsquo;s start with the &ldquo;Kleisli Arrows&rdquo;, which are monotone functions \(f : A \to P(B) \) where \(A,B \) are posets and \(P(B)\) represents the poset of downward-closed subsets of \(B \).</p> + +<p>Now to &ldquo;de-posetize&rdquo; this, we&rsquo;ll take sets \(X,Y \) and let \(P(Y) \) mean the powerset of \(Y\), that is the set of all subsets of \(Y \). Then a function \(f : X \to P(Y) \) is actually exactly the same thing as a relation \(R \subset X \times Y \). From \(f : +X \to P(Y) \) we can take \(R = \{(x,y) \in X\times Y | y\in f(x)\} \) and from \(R\) we can construct \(f(x) = \{y \in Y | (x,y) \in R \}\).</p> + +<p>Furthermore, the &ldquo;Kleisli composition&rdquo; is the same as composition of relations. If \(R \subset X \times Y \) and \(Q \subset Y \times Z +\), then the composition is defined as \[ (R;Q) = \{(x,z) \in X \times Z | \exists y\in Y. (x,y) \in R \land (y,z) \in Q\}\]</p> + +<p>Then the next thing we need to understand is what is the de-generalization of &ldquo;Kleisli Galois connection&rdquo;? Well, Galois connections are an instance of what&rsquo;s called an adjunction in category theory, which is usually formulated in terms of categories, functors and natural transformations. However, you can interpret the definition of adjunction in any &ldquo;universe&rdquo; that acts like the universe of categories, functors and natural transformations and it turns out we have such a universe. The universe I&rsquo;m talking about is called \(\texttt{Rel}\), and it consists of sets, relations between sets and <em>inclusion of relations</em>, i.e. that one relation is a subset of another.</p> + +<p>Then what does it mean to have an adjunction between two relations \(R \subset X \times Y, Q \subset Y \times X\)? Taking apart the definition it just means</p> + +<p>\begin{align}\tag{1} \Delta(X) \subset R;Q \end{align} \begin{align}\tag{2} Q;R \subset \Delta(Y) \end{align}</p> + +<p>where \(\Delta \) means the <em>diagonal</em>, or equality relation on the set:</p> + +<p>\[\Delta(X) = \{(x_1,x_2) \in X | x_1 = x_2 \} \]</p> + +<p>So we just need to unravel what (1) and (2) mean above. Unwinding (1), we get that for any \(x \in X\), there exists a \(y \in Y \) such that \((x,y) \in R \) and \((y,x) \in Q\). This tells us for one that \(R \) is a &ldquo;right-total&rdquo; relation and \(Q \) is a &ldquo;left-total&rdquo; relation. Every \(x \) is related to some \( y\) by \( R \) and \( Q\).</p> + +<p>If we unwind (2), we get that for any \(y,y' \in Y\) if there&rsquo;s some \(x \in X \) such that \((x,y) \in R \) and \((y',x) \in Q \) then actually \(y = y')\). This one is a bit more mysterious, but first, let&rsquo;s see what this tells us about the relationship between \(R\) and \(Q \).</p> + +<p>If \((x,y) \in R \), then by (1) there&rsquo;s some \(y' \in Y\) so that \((x,y') \in R \) and \((y',x) \in Q\). Then, by (2) we know that \(y = y'\), so we&rsquo;ve shown that if \((x,y) \in R \) then \((y,x) +\in Q\). Then a completely symmetric argument shows that if \((y,x) +\in Q \) then \((x,y)\in R\)! So we&rsquo;ve discovered that actually \(Q \) is just the opposite relation of \(R \).</p> + +<p>Then if we look at (2) again but replace the \(Q\)&rsquo;s by flipped \(R\)&rsquo;s we get that for any \(y,y' \in Y\), if there&rsquo;s some \(x +\in X\) such that \((x,y) \in R \) and \((x,y')\in R\) then \(y += y'\), which tells us that \(R \) is a partial function, i.e., that every \(x \) is related to at most one \(y \) by \(R \).</p> + +<p>You may recognize it now, our \(R \subset X \times Y \) is just a function, and saying \(R, Q\) are adjoint is exactly the same as saying that \(Q = R^{\text{op}}\) and \(R \) is a function. Adjunctions are so pervasive you saw them back in pre-algebra!</p> + +<h1 id="constructive-galois-connections">Constructive Galois Connections</h1> + +<p>Back to constructive Galois connections, I hope if you read the paper you can see that their theorem is a generalization of the above argument, where instead of relations we have &ldquo;monotone relations&rdquo;, i.e., downward-closed \(R \subset A^{\text{op}} \times B \). Then you can interpret the definition of adjunction in that universe and get that it&rsquo;s the same as a Kleisli Galois connection and that a similar argument to the above shows that the &ldquo;left adjoint&rdquo; is represented by a monotone function \(f : A \to B \):</p> + +<p>\[R = \{(x,y) | y \le f(x) \} \]</p> + +<p>Which shows that every Kleisli Galois connection is actually a constructive Galois connection! The details are in their paper, and I hope they are easier to follow now.</p> + +<p>In fact, we get a little extra from what&rsquo;s mentioned in their paper, which is that the &ldquo;right adjoint&rdquo; is represented by \(f \) as well but in the opposite way:</p> + +<p>\[Q = \{(y,x) | f(x) \le y \}\]</p> + +<h1 id="category-theory-post-scriptum">Category Theory Post Scriptum</h1> + +<p>If you&rsquo;re interested in Category theory, here&rsquo;s a more technical addendum.</p> + +<p>Remembering from Category Theory class, sets are just posets where objects are only less than themselves and posets are (basically) categories where there is at most 1 arrow between objects, so we might naturally ask, does this theorem extend to categories?</p> + +<p>Well, first we need a generalization from relations to downward-closed relations to what are called <a href="https://ncatlab.org/nlab/show/profunctor">distributors or profunctors</a>. Then we can also generalize inclusion of relations to morphisms of distributors and ask, is every left adjoint distributor represented by a functor?</p> + +<p>The answer is, at least in full generality, no! For it to be true we need a special property on the codomain of the left adjoint \(R : C +\not\to D \), which is called (for mind-boggling reasons) <a href="https://ncatlab.org/nlab/show/Cauchy+complete+category#InOrdinaryCatTheoryByProfunctors">Cauchy completeness</a>. Viewing sets and posets as special categories, it turns out that they always have this property, and that&rsquo;s why the theorem worked out for those adjunctions.</p> \ No newline at end of file diff --git a/blog/feeds/migratory-typing.atom.xml b/blog/feeds/migratory-typing.atom.xml new file mode 100644 index 00000000..700d47b3 --- /dev/null +++ b/blog/feeds/migratory-typing.atom.xml @@ -0,0 +1,874 @@ + + + PRL Blog: Posts tagged 'migratory typing' + + + urn:http-prl-ccs-neu-edu:-blog-tags-migratory-typing-html + 2020-12-23T18:21:55Z + + Deep and Shallow Types + + urn:http-prl-ccs-neu-edu:-blog-2020-12-23-deep-and-shallow-types + 2020-12-23T18:21:55Z + 2020-12-23T18:21:55Z + + Ben Greenman + +<p>I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:</p> + +<ul> + <li><a href="http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020</a></li></ul> + +<p>To the PRL: thanks for a wonderful 6.5 years.</p> +<!-- more--> + +<h3 id="abstract">Abstract</h3> + +<blockquote> + <p>The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound&rsquo;&rsquo; mixed-typed languages from others that enforce type obligations locally rather than globally.</p> + <p>In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.</p> + <p>My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.</p> + <p>I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.</p></blockquote> + +<p>Next year, I&rsquo;ll be a <a href="https://cifellows2020.org">CI Fellow</a> at Brown.</p> + + Complete Monitors for Gradual Types + + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + 2019-10-31T21:58:26Z + 2019-10-31T21:58:26Z + + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + Forgetful and Heedful contracts + + urn:http-prl-ccs-neu-edu:-blog-2019-04-07-forgetful-and-heedful-contracts + 2019-04-07T23:15:11Z + 2019-04-07T23:15:11Z + + Ben Greenman + +<p><em>Forgetful</em> and <em>heedful</em> are two methods for space-efficient contracts developed by <a href="http://www.cs.pomona.edu/~michael/">Michael Greenberg</a> in <a href="https://arxiv.org/abs/1410.2813">2014</a>. These methods were born in the shadow of a third method, <em>eidetic</em>, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.</p> +<!-- more--> + +<hr /> + +<p>Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of <a href="https://www2.ccs.neu.edu/racket/pubs/popl11-dfff.pdf">three parties</a>: the API code, the client code, and the contract between them.</p> + +<p>For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:</p> + +<pre><code> #lang racket/base + + (module sort racket + (provide + (contract-out + [quicksort + (-&gt; (vectorof point/c) void?)])) + + (define point/c (vectorof integer?)) + + (define (quicksort points) + ....)) + + (module client racket + (require (submod ".." sort)) + (quicksort '())) + + (require 'client)</code></pre> + +<pre><code>quicksort: contract violation; + expected a vector + given: '() + in: the 1st argument of + (-&gt; (vectorof (vectorof integer?)) void?) + contract from: + (file.rkt sort) + blaming: (file.rkt client) + (assuming the contract is correct)</code></pre> + +<p>That covers the basics. For an extended introduction to contracts, visit <a href="https://docs.racket-lang.org/guide/contracts.html">The Racket Guide</a>.</p> + +<p>The quicksort example and the related figures are from the paper <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts: Fixing a Pathology of Gradual Typing</em></a></p> + +<h3 id="classic-contracts-and-space-efficiency">Classic contracts and &ldquo;Space Efficiency&rdquo;</h3> + +<p>The <code>(vectorof point/c)</code> contract used above describes a possibly-mutable array whose elements match the <code>point/c</code> contract. Since the array can be mutated, this contract has implications for two parties:</p> + +<ol> + <li>the client module must supply a good array, and</li> + <li>the sorting module must not insert a bad element.</li></ol> + +<p>To enforce the second condition, the <code>vectorof</code> contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:</p> + +<pre><code>(quicksort (vector (vector 4 4) + (vector 2 2) + (vector 1 1) + (vector 3 3)))</code></pre> + +<p>After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a <strong>solid</strong> blue bar for the <strong>write</strong> checks against the sort module and a <em>striped</em> blue bar for the <em>read</em> checks against the client.</p> + +<p><img src="/img/vector-chaperone-0.png" alt="A wrapped vector" /></p> + +<p>In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted &mdash; because every <strong>write</strong> and <em>read</em> applies a contract to the element.</p> + +<p><img src="/img/vector-chaperone-1.png" alt="Layers of element wrappers" /></p> + +<p>On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.</p> + +<p>Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and &ldquo;indirecting&rdquo; future writes/reads through wrappers. These space and time costs can add up.</p> + +<blockquote> + <p>&ldquo;on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times&rdquo; &mdash; <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts</em></a></p></blockquote> + +<p>To fix the problem, researchers have been exploring <em>space-efficient</em> implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and <em>eidetic</em>, <em>forgetful</em>, and <em>heedful</em> are his names for three implementations.</p> + +<p>(Although the goal of this post is to promote <em>forgetful</em> and <em>heedful</em>, we will review all three.)</p> + +<h3 id="eidetic-space-efficiency">Eidetic space-efficiency</h3> + +<p>The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a <em>merge</em> operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal &ldquo;wrapping&rdquo; contracts and their size is bounded by the number (and height) of source-code contracts in the program.</p> + +<p>An eidetic contract is an <code>N</code>-ary tree (for <code>N &gt; 0</code>):</p> + +<ul> + <li>each node represents a higher-order contract combinator, such as <code>vectorof</code></li> + <li>the <code>N</code> children of a node represent the different interactions that the value supports</li> + <li>each leaf is a list of non-higher-order, or <em>flat</em>, contracts</li></ul> + +<p>For example, the <code>(vectorof point/c)</code> source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf">Collapsible Contracts</a> paper has an illustration. Each tree node represents a <code>vectorof</code> contract; these nodes have <code>N=2</code> children because vectors support reads and writes.</p> + +<p>A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists &mdash; if an implementation comes with a <code>contract-stronger?</code> predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.</p> + +<p>Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should <code>contract-stronger?</code> try to solve problems that lack polynomial-time solutions?</p> + +<p>Thankfully, there are at least two &ldquo;compromise&rdquo; alternatives.</p> + +<h3 id="forgetful-space-efficiency">Forgetful space-efficiency</h3> +<!-- "no operation relies on e being a T2, skipping the check doesn't risk soundness" p.12--> +<!-- "In forgetful \lambda_H, we offer a simple solution to space inefficient casts: just forget about them" p.11--> +<!-- "Just the same, when accumulating casts on the stack, we throw away all but the last cast" p.11--> +<!-- "forgetful ... skip[s] checks and change[s] blame labels" p.3--> + +<blockquote> + <p>&ldquo;Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 10, bottom of page 23--></blockquote> + +<p>The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:</p> + +<ol> + <li>the client obligations from the old contract, and</li> + <li>the server obligations from the new contract</li></ol> + +<p>If not, proceed as usual &mdash; by wrapping (an unwrapped value) or raising an error. Every value receives at most <strong>one</strong> wrapper; this wrapper changes as the value flows to different clients.</p> + +<p>Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module <code>A</code> exports a function <code>f</code> with contract <code>(-&gt; T1 T2)</code> to module <code>B</code>, and suppose module <code>B</code> shares this function with a few other client modules using different contracts. As <code>f</code> flows to a new client, it keeps the <code>T1</code> domain check and gets a replacement for the <code>T2</code> codomain check.</p> + +<ul> + <li>Keeping <code>T1</code> ensures that the code inside the function (defined by module <code>A</code>) receives input that matches its expectation.</li> + <li>Replacing <code>T2</code> ensures that each new client receives output that it expects.</li></ul> + +<p>Unfortunately, replacing <code>T2</code> also means that clients of module <code>B</code> cannot trust the <code>T2</code> contract. This contract is not checked, and so forgetful contracts <strong>miss</strong> some errors that would be caught by standard contracts. For the same reason, a bug in module <code>B</code> may go undetected by its clients &mdash; even if a later contract reports an issue, the contract system has no memory that <code>B</code> was partly-responsible.</p> + +<p>Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.</p> + +<h3 id="heedful-space-efficiency">Heedful space-efficiency</h3> + +<p>A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set &mdash; or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.</p> + +<p>To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.</p> + +<p>The heedful method is a compromise between forgetful and eidetic.</p> + +<ul> + <li> + <p>Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of <code>contract-stronger?</code> predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.</p></li> + <li> + <p>Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.</p></li></ul> + +<p>For details, see <a href="https://arxiv.org/abs/1410.2813">the extended version</a> of Michael&rsquo;s POPL 2015 paper. Don&rsquo;t bother searching <a href="http://www.cs.pomona.edu/~michael/papers/popl2015_space.pdf">the conference version</a> &mdash; aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.</p> + +<p><code>*</code> If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to <a href="http://mballantyne.net/">Michael Ballantyne</a> for explaining this to me.</p> + +<h3 id="priorities-and-appearances">Priorities and Appearances</h3> + +<p>The extended version of <em>Space-Efficient Manifest Contracts</em> introduces the forgetful and heedful methods with extreme modesty. It&rsquo;s tempting to skip past them and focus on the eidetic method.</p> + +<blockquote> + <p>&ldquo;Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [<a href="https://dl.acm.org/citation.cfm?doid=1706299.1706342">Siek and Wadler 2010</a>]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen&mdash;warm up exercises for eidetic.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 1, bottom of page 2--></blockquote> + +<p>And yet, at least two other research papers rely on these &ldquo;strawmen&rdquo; &mdash; or rather, the ideas behind the names.</p> + +<p><a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a>, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.</p> + +<blockquote> + <p>&ldquo;if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- page 21--></blockquote> + +<p>This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.</p> + +<blockquote> + <p>&ldquo;removing these casts preserves the soundness of the evaluation while reducing the number of them&rdquo;</p> + <p>&ldquo;while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- p.27--><!-- page 21--></blockquote> +<!-- The followup at POPL 2019 is not forgetful.--> +<!-- It's similar to eager coercions ... keep all types around and error--> +<!-- if there's a new type that doesn't match the old ones.--> +<!-- Also, that paper chooses not to let functions have intersection types,--> +<!-- which kind-of-avoids the questions ... but really the eagerness is key.--> + +<p><a href="https://dl.acm.org/citation.cfm?id=3009849"><em>Big Types in Little Runtime</em></a>, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their <em>transient</em> semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.</p> + +<p>That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value&rsquo;s set. When a mismatch occurs, the semantics <a href="https://www.ccs.neu.edu/home/types/resources/notes/transient-undefined-blame-extract.pdf">tries to report</a> every type in the set that relates to the mismatch.</p> + +<p>And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">ICFP 2018</a>. Where will we see them next?</p> + + The Behavior of Gradual Types: A User Study + + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + 2018-12-11T19:50:33Z + 2018-12-11T19:50:33Z + + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + Java and Migratory Typing + + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + 2018-12-02T14:41:53Z + 2018-12-02T14:41:53Z + + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> + + A Spectrum of Type Soundness and Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-10-06-a-spectrum-of-type-soundness-and-performance + 2018-10-06T11:23:35Z + 2018-10-06T11:23:35Z + + Ben Greenman + +<p>The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>A Spectrum of Type Soundness and Performance</em> by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018">http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018</a></p></blockquote> + +<p>A dynamically-typed language runs any program that &ldquo;looks good&rdquo; (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both &ldquo;looks good&rdquo; and is well-typed according to a type checker.</p> + +<p>A <em>mixed-typed</em> language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.</p> + +<div class="figure"><img src="/img/mixed-typed-systems-by-year.png" alt="Figure 1: Some mixed-typed languages" /> + <p class="caption">Figure 1: Some mixed-typed languages</p></div> + +<p>These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:</p> + +<ul> + <li><strong>MACLISP</strong> defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the <a href="http://www.softwarepreservation.org/projects/LISP/MIT/Moon-MACLISP_Reference_Manual-Apr_08_1974.pdf">Moonual</a>. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).</li> + <li><strong>Strongtalk</strong> includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a> position paper.</li> + <li><strong>Typed Racket</strong> lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.</li> + <li><strong>Thorn</strong> requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.</li> + <li><strong>Reticulated</strong> lets a program combine static and dynamic <em>expressions</em> and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek <a href="https://dl.acm.org/citation.cfm?id=3009849">POPL 2017</a>).</li></ul> + +<p>That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system&rsquo;s protocol for checking dynamically-typed values at a <em>type boundary</em> (between static and dynamic code).</p> + +<h3 id="main-contribution">Main Contribution</h3> + +<p>In the paper <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>A Spectrum of Type Soundness and Performance</em></a>, we define a tiny mixed-typed language and show three ways to define the behavior of programs &mdash; based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.</p> + +<p>The three behaviors are inspired by existing languages. A <strong>higher order</strong> behavior ensures that dynamically-typed values match the static type at a boundary &mdash; by checking the value when possible, and by monitoring the value&rsquo;s future interactions when necessary. A <strong>first order</strong> behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An <strong>erasure</strong> behavior does no checking whatsoever.</p> + +<blockquote> + <p>Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function <code>f</code>, then one way to enforce the type boundary is to wrap <code>f</code> in a proxy to assert that every future call to <code>f</code> returns a number. In this case, the proxy monitors the behavior of <code>f</code>.</p></blockquote> + +<p>Concretely, the paper defines three formal semantics with the same names. The <strong>higher-order</strong> semantics enforces full types at the boundaries (Section 2.3). The <strong>first-order</strong> semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every &ldquo;selector&rdquo; operation in typed code, e.g., function application, data structure access (Section 2.5). The <strong>erasure</strong> semantics simply ignores the types (Section 2.4).</p> + +<p>Each semantics satisfies a <em>different</em> notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">online supplement</a> gives full proofs.</p> + +<p>The paper has more to say about the meta-theory. See section 2 and section 4.</p> + +<blockquote> + <p>To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for <a href="https://dl.acm.org/citation.cfm?id=2676971">subset of the language</a> (in the spirit of <a href="http://soundiness.org/">soundiness</a>) or use the name &ldquo;type soundness&rdquo; to describe <a href="https://dl.acm.org/citation.cfm?id=2676971">a different property</a>.</p></blockquote> + +<p>Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations &mdash; similar to Typed Racket&rsquo;s <a href="http://docs.racket-lang.org/ts-reference/Typed_Racket_Syntax_Without_Type_Checking.html">no-check</a> language.</p> + +<p>Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (<a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016</a>). The programs we measured are the non-object-oriented ones from our <a href="http://docs.racket-lang.org/gtp-benchmarks/index.html">benchmark suite</a>.</p> + +<p>To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises &mdash; see section 3 of the paper, section B of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, and/or the <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slides</a>.</p> + +<h3 id="implications">Implications</h3> + +<ol> + <li>The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, or <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slide 114</a>.</li> + <li>Programmers using mixed-typed languages need to know what guarantees their types provide. (It is <a href="https://twitter.com/jbandi/status/965005464638541825">not safe to assume that TypeScript types give the same guarantees as OCaml types</a>!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.</li> + <li>The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?</li></ol> + +<blockquote> + <p>Note: in this post, a <em>mixed-typed language</em> is one that allows any combination of static and dynamic typing. A <em>gradually-typed language</em> is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>).</p></blockquote> + + Sampling Gradual Typing Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + 2018-05-08T15:37:37Z + 2018-05-08T15:37:37Z + Ben Greenman + Zeina Migeed + + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> \ No newline at end of file diff --git a/blog/feeds/migratory-typing.rss.xml b/blog/feeds/migratory-typing.rss.xml new file mode 100644 index 00000000..e5530338 --- /dev/null +++ b/blog/feeds/migratory-typing.rss.xml @@ -0,0 +1,860 @@ + + + + PRL Blog: Posts tagged 'migratory typing' + PRL Blog: Posts tagged 'migratory typing' + http://prl.ccs.neu.edu/blog/tags/migratory-typing.html + Wed, 23 Dec 2020 18:21:55 UT + Wed, 23 Dec 2020 18:21:55 UT + 1800 + + Deep and Shallow Types + http://prl.ccs.neu.edu/blog/2020/12/23/deep-and-shallow-types/?utm_source=migratory-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-12-23-deep-and-shallow-types + Wed, 23 Dec 2020 18:21:55 UT + Ben Greenman + +<p>I successfully defended my Ph.D. dissertation. You can find the document, a talk recording, and much more here:</p> + +<ul> + <li><a href="http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">http://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020</a></li></ul> + +<p>To the PRL: thanks for a wonderful 6.5 years.</p> +<!-- more--> + +<h3 id="abstract">Abstract</h3> + +<blockquote> + <p>The design space of mixed-typed languages is lively but disorganized. On one hand, researchers across academia and industry have contributed language designs that allow typed code to interoperate with untyped code. These design efforts explore a range of goals; some improve the expressiveness of a typed language, and others strengthen untyped code with a tailor-made type system. On the other hand, experience with type-sound designs has revealed major challenges. We do not know how to measure the performance costs of sound interaction. Nor do we have criteria that distinguish ``truly sound&rsquo;&rsquo; mixed-typed languages from others that enforce type obligations locally rather than globally.</p> + <p>In this dissertation, I introduce methods for assessing mixed-typed languages and bring order to the design space. My first contribution is a performance-analysis method that allows language implementors to systematically measure the cost of mixed-typed interaction.</p> + <p>My second contribution is a design-analysis method that allows language designers to understand implications of the type system. The method addresses two central questions: whether typed code can cope with untyped values, and whether untyped code can trust static types. Further distinctions arise by asking whether error outputs can direct a programmer to potentially-faulty interactions.</p> + <p>I apply the methods to several designs and discover limitations that motivate a synthesis of two ideas from the literature: deep types and shallow types. Deep types offer strong guarantees but impose a high interaction cost. Shallow types offer weak guarantees and better worst-case costs. This dissertation proves that deep and shallow types can interoperate and measures the benefits of a three-way mix.</p></blockquote> + +<p>Next year, I&rsquo;ll be a <a href="https://cifellows2020.org">CI Fellow</a> at Brown.</p> + + Complete Monitors for Gradual Types + http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/?utm_source=migratory-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-10-31-complete-monitors-for-gradual-types + Thu, 31 Oct 2019 21:58:26 UT + Ben Greenman + +<p>Syntactic type soundness is too weak to tell apart different ways of running a program that mixes typed and untyped code. Complete monitoring is a stronger property that captures a meaningful distinction &mdash; a language satisfies complete monitoring iff it checks all interactions between typed and untyped code.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>Complete Monitors for Gradual Types</em> by Ben Greenman, Matthias Felleisen, and Christos Dimoulas. For the full paper, proofs, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gfd-oopsla-2019">click here</a>.</p></blockquote> + +<h3 id="example-clickable-plot">Example: Clickable Plot</h3> + +<p>The program below has a subtle bug. Can you find it?</p> + +<p><img src="/img/complete-monitoring-0.png" alt="Untyped client code, a typed API, and untyped library code." /></p> + +<p>First of all, this pseudocode program combines three chunks of code:</p> + +<ul> + <li> + <p>On the left, an <strong>untyped</strong> client script defines a function <code>h</code> that expects a pair of numbers and returns an image. The client uses this function to create a <code>ClickPlot</code> object, and then displays the plot &mdash; ideally in a new GUI window.</p></li> + <li> + <p>In the center, a <strong>typed</strong> API file describes a <code>ClickPlot</code> object as something with one constructor and two methods. The constructor expects a function; according to the type, such functions can expect a pair of numbers and must compute an image. The <code>mouseHandler</code> method expects a <code>MouseEvt</code> object and returns nothing. The <code>show</code> method expects no arguments and returns nothing. (Presumably, these methods have side effects.)</p></li> + <li> + <p>On the right, an <strong>untyped</strong> library module implements a <code>ClickPlot</code> object. Most of the code is omitted (<code>...</code>), but the <code>mouseHandler</code> method sends its input directly to the <code>onClick</code> callback.</p></li></ul> + +<p>The <strong>bug</strong> is in the API &mdash; in the type <code>([N, N]) =&gt; Image</code>. This type promises that a given function can expect a pair of numbers, and indeed the client function <code>h</code> expects a pair. But the library code on the right sends a <code>MouseEvt</code> object.</p> + +<p>What happens when we run this program in a type-sound mixed-typed language? Does <code>h</code> receive the invalid input?</p> + +<p>As it turns out, type soundness cannot say. A type sound language may choose to enforce or ignore the fact that the API promises a pair of numbers to the client.</p> + +<h3 id="type-soundness-is-not-enough">Type Soundness is Not Enough</h3> + +<p>Sound types are statements about the behavior of a program. A normal type soundness theorem for a typed language says that a well-typed program can either compute a value of the same type, compute forever (diverge), or stop with an acceptable error (perhaps division by zero). No other behaviors are possible.</p> + +<blockquote> + <p><strong>Classic Type Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>A mixed-typed language needs two &ldquo;type soundness&rdquo; theorems: one for typed code and one for untyped code. The <strong>typed</strong> soundness theorem can resemble a classic theorem. The <strong>untyped</strong> soundness theorem is necessarily a weaker statement due to the lack of types:</p> + +<blockquote> + <p><strong>Mixed-Typed Soundness</strong></p> + <p>If <code>e : T</code> then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v : T</code></li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul> + <p>And if <code>e</code> is untyped then one of the following holds:</p> + <ul> + <li><code>e --&gt;* v</code> and <code>v</code> is an untyped value</li> + <li><code>e</code> diverges</li> + <li><code>e --&gt;* OkError</code></li></ul></blockquote> + +<p>Now we can see why mixed-typed soundness is not strong enough to guarantee that the callback <code>h</code> in the code above receives a pair value. We have an <strong>untyped</strong> function called from an <strong>untyped</strong> context &mdash; since there are no types sitting right there, type soundness has nothing to say except that the untyped code can expect an untyped value!</p> + +<p><img height="200px" src="/img/complete-monitoring-1.png" alt="Untyped library sends input directly to untyped client." /></p> + +<p>Nevertheless, this channel of communication between the library and client arose through the typed API. One might expect the type <code>[N, N]</code> to restrict the values that can flow across the channel; indeed, if types really are statements about the behavior of a program, then the channel needs to be protected.</p> + +<p>The question is: what formal property separates languages thet check all typed/untyped channels of communication (whether direct or derived)? One answer is complete monitoring.</p> + +<h3 id="complete-monitoring">Complete Monitoring</h3> + +<p>A mixed-typed language satisfies complete monitoring iff evaluation never lets a value flow un-checked across a type boundary. To make this idea precise, we need to enrich the syntax of the language with a specification of <em>ownership</em> to say what parts of the program are responsible for different values, and to say how evalution changes responsibilities. Relative to a specification, complete monitoring states that every expression that arises during evaluation is made up of parts that each have a single owner.</p> + +<blockquote> + <p><em>Complete Monitoring</em></p> + <p>For all well-formed <code>e</code> and all <code>e'</code>, if <code>e --&gt;* e'</code> then every subexpression of <code>e'</code> has a unique owner.</p></blockquote> + +<p>This property separates our two behaviors for the Clickable Plot code. A language that satisfies complete monitoring enforces the API types with a runtime check. A language that merely satisfies type soundness may skip these checks.</p> + +<h3 id="an-aid-to-debugging">An Aid to Debugging</h3> + +<p>The question raised by the Clickable Plot example is whether a language can <strong>detect</strong> one mismatch between a type and a value. A language that satisfies complete monitoring detects all such mis-matches. But we can say more. If a mismatch occurs, then programmer knows exactly where to start debugging &mdash; either the type is an incorrect specification, or the given value is flawed. In other words, complete monitoring implies a concise 2-party explanation for every type mismatch.</p> + +<p>The paper generalizes this goal of explaining a mismatch for languages that fail to satisfy complete monitoring. There may be 2N parties to blame thanks to un-checked channels of communication, and we want to be certain to report all these parties and no false positives.</p> + +<p>Also in the paper, you can find:</p> + +<ul> + <li>a model of ownership, clear <em>laws</em> for how ownership changes during evaluation;</li> + <li>examples of how to systematically add ownership to an operational semantics to attempt a proof of complete monitoring;</li> + <li>definitions for <strong>blame soundness</strong> and <strong>blame completeness</strong>;</li> + <li>an analysis of three semantics, which correspond to <a href="https://docs.racket-lang.org/ts-reference/index.html">Typed Racket</a>, <a href="http://hdl.handle.net/2022/23172">Transient Reticulated</a>, and a compromise;</li> + <li>and discussion of an alternative, heap-based model of ownership.</li></ul> + +<p>Paper: <a href="https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf">https://www2.ccs.neu.edu/racket/pubs/oopsla19-gfd.pdf</a></p> + + Forgetful and Heedful contracts + http://prl.ccs.neu.edu/blog/2019/04/07/forgetful-and-heedful-contracts/?utm_source=migratory-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-04-07-forgetful-and-heedful-contracts + Sun, 07 Apr 2019 23:15:11 UT + Ben Greenman + +<p><em>Forgetful</em> and <em>heedful</em> are two methods for space-efficient contracts developed by <a href="http://www.cs.pomona.edu/~michael/">Michael Greenberg</a> in <a href="https://arxiv.org/abs/1410.2813">2014</a>. These methods were born in the shadow of a third method, <em>eidetic</em>, with stronger theoretic properties. Since then, however, the forgetful method has been re-invented at least twice. Both deserve a second look.</p> +<!-- more--> + +<hr /> + +<p>Contracts are a tool for specifying and dynamically-enforcing the behavior of a program. In a language with contracts, a programmer can annotate an API with code that documents the intended use for other readers. When client code interacts with such an API, the annotations ensure that the actual behavior matches the expected. If there is a mismatch, the contract annotations can report an issue in terms of <a href="https://www2.ccs.neu.edu/racket/pubs/popl11-dfff.pdf">three parties</a>: the API code, the client code, and the contract between them.</p> + +<p>For example, a Racket module that exports a sorting function can use a contract to describe the kind of input it expects. If a client module sends invalid input, the contract blames the client module for the error, assuming that the contract is bug-free:</p> + +<pre><code> #lang racket/base + + (module sort racket + (provide + (contract-out + [quicksort + (-&gt; (vectorof point/c) void?)])) + + (define point/c (vectorof integer?)) + + (define (quicksort points) + ....)) + + (module client racket + (require (submod ".." sort)) + (quicksort '())) + + (require 'client)</code></pre> + +<pre><code>quicksort: contract violation; + expected a vector + given: '() + in: the 1st argument of + (-&gt; (vectorof (vectorof integer?)) void?) + contract from: + (file.rkt sort) + blaming: (file.rkt client) + (assuming the contract is correct)</code></pre> + +<p>That covers the basics. For an extended introduction to contracts, visit <a href="https://docs.racket-lang.org/guide/contracts.html">The Racket Guide</a>.</p> + +<p>The quicksort example and the related figures are from the paper <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts: Fixing a Pathology of Gradual Typing</em></a></p> + +<h3 id="classic-contracts-and-space-efficiency">Classic contracts and &ldquo;Space Efficiency&rdquo;</h3> + +<p>The <code>(vectorof point/c)</code> contract used above describes a possibly-mutable array whose elements match the <code>point/c</code> contract. Since the array can be mutated, this contract has implications for two parties:</p> + +<ol> + <li>the client module must supply a good array, and</li> + <li>the sorting module must not insert a bad element.</li></ol> + +<p>To enforce the second condition, the <code>vectorof</code> contract wraps incoming vectors in a proxy that checks future writes. Suppose the client sends a vector with four points:</p> + +<pre><code>(quicksort (vector (vector 4 4) + (vector 2 2) + (vector 1 1) + (vector 3 3)))</code></pre> + +<p>After applying the contract, the vector is wrapped in a proxy that checks incoming writes and outgoing reads. The following picture illustrates the wrapper with a <strong>solid</strong> blue bar for the <strong>write</strong> checks against the sort module and a <em>striped</em> blue bar for the <em>read</em> checks against the client.</p> + +<p><img src="/img/vector-chaperone-0.png" alt="A wrapped vector" /></p> + +<p>In a straightforward implementation, these wrappers can stack up if multiple contracts are applied to the same value. For our quicksort in particular, the elements of the vector are mutable vectors and may accumulate wrappers as the vector is sorted &mdash; because every <strong>write</strong> and <em>read</em> applies a contract to the element.</p> + +<p><img src="/img/vector-chaperone-1.png" alt="Layers of element wrappers" /></p> + +<p>On the bright side, these wrappers enforce the contracts and help the programmer understand the source of the error if any contract is violated.</p> + +<p>Unfortunately, the wrappers also affect the performance of the program. There are prices to pay for: (1) checking values against the contracts, (2) allocating new wrappers, (3) and &ldquo;indirecting&rdquo; future writes/reads through wrappers. These space and time costs can add up.</p> + +<blockquote> + <p>&ldquo;on a randomly ordered vector of 1,000 points, a call to quicksort can wrap the inner vectors an average of 21 times&rdquo; &mdash; <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf"><em>Collapsible Contracts</em></a></p></blockquote> + +<p>To fix the problem, researchers have been exploring <em>space-efficient</em> implementations of contracts that attach a bounded number of wrappers to any value. Michael Greenberg is one of these researchers, and <em>eidetic</em>, <em>forgetful</em>, and <em>heedful</em> are his names for three implementations.</p> + +<p>(Although the goal of this post is to promote <em>forgetful</em> and <em>heedful</em>, we will review all three.)</p> + +<h3 id="eidetic-space-efficiency">Eidetic space-efficiency</h3> + +<p>The eidetic method introduces a data structure to represent higher-order contracts. The structure supports a <em>merge</em> operation; when two contracts meet, they are merged in a way that avoids duplication. Eidetic contracts have the same behavior as normal &ldquo;wrapping&rdquo; contracts and their size is bounded by the number (and height) of source-code contracts in the program.</p> + +<p>An eidetic contract is an <code>N</code>-ary tree (for <code>N &gt; 0</code>):</p> + +<ul> + <li>each node represents a higher-order contract combinator, such as <code>vectorof</code></li> + <li>the <code>N</code> children of a node represent the different interactions that the value supports</li> + <li>each leaf is a list of non-higher-order, or <em>flat</em>, contracts</li></ul> + +<p>For example, the <code>(vectorof point/c)</code> source-code contract describes an eidetic tree with 3 nodes and 4 singleton-list leaves. Section 3.1 of the <a href="http://users.cs.northwestern.edu/~robby/pubs/papers/oopsla2018-fgsfs.pdf">Collapsible Contracts</a> paper has an illustration. Each tree node represents a <code>vectorof</code> contract; these nodes have <code>N=2</code> children because vectors support reads and writes.</p> + +<p>A successful merge combines two trees of the same shape by re-using half the nodes and appending the leaf lists. Re-using nodes saves some space, and helps reduce the overhead of trees relative to simple wrapping contracts. The main savings comes from filtering the leaf lists &mdash; if an implementation comes with a <code>contract-stronger?</code> predicate that tests whether one flat contract accepts fewer values than a second, then it can remove leaf-list contracts that are preceded by stronger ones. Trees make this filtering possible.</p> + +<p>Suffice to say, eidetic is an ideal solution in theory but comes with practical challenges. Are trees more expensive than wrappers in the common case? Can the leaf-lists in a tree share elements? Should <code>contract-stronger?</code> try to solve problems that lack polynomial-time solutions?</p> + +<p>Thankfully, there are at least two &ldquo;compromise&rdquo; alternatives.</p> + +<h3 id="forgetful-space-efficiency">Forgetful space-efficiency</h3> +<!-- "no operation relies on e being a T2, skipping the check doesn't risk soundness" p.12--> +<!-- "In forgetful \lambda_H, we offer a simple solution to space inefficient casts: just forget about them" p.11--> +<!-- "Just the same, when accumulating casts on the stack, we throw away all but the last cast" p.11--> +<!-- "forgetful ... skip[s] checks and change[s] blame labels" p.3--> + +<blockquote> + <p>&ldquo;Forgetful is an interesting middle ground: if contracts exist to make partial operations safe (and not abstraction or information hiding), forgetfulness may be a good strategy.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 10, bottom of page 23--></blockquote> + +<p>The forgetful method is exceptionally simple. When applying a new contract to a value, first check whether it is wrapped in a similar contract. If so, then replace the existing wrapper with one that combines:</p> + +<ol> + <li>the client obligations from the old contract, and</li> + <li>the server obligations from the new contract</li></ol> + +<p>If not, proceed as usual &mdash; by wrapping (an unwrapped value) or raising an error. Every value receives at most <strong>one</strong> wrapper; this wrapper changes as the value flows to different clients.</p> + +<p>Forgetful is safe in the sense that every piece of code can trust the top-level shape of the values it receives. Suppose module <code>A</code> exports a function <code>f</code> with contract <code>(-&gt; T1 T2)</code> to module <code>B</code>, and suppose module <code>B</code> shares this function with a few other client modules using different contracts. As <code>f</code> flows to a new client, it keeps the <code>T1</code> domain check and gets a replacement for the <code>T2</code> codomain check.</p> + +<ul> + <li>Keeping <code>T1</code> ensures that the code inside the function (defined by module <code>A</code>) receives input that matches its expectation.</li> + <li>Replacing <code>T2</code> ensures that each new client receives output that it expects.</li></ul> + +<p>Unfortunately, replacing <code>T2</code> also means that clients of module <code>B</code> cannot trust the <code>T2</code> contract. This contract is not checked, and so forgetful contracts <strong>miss</strong> some errors that would be caught by standard contracts. For the same reason, a bug in module <code>B</code> may go undetected by its clients &mdash; even if a later contract reports an issue, the contract system has no memory that <code>B</code> was partly-responsible.</p> + +<p>Despite these changes in behavior, forgetful is a straightforward method for saving space and time relative to classic contracts.</p> + +<h3 id="heedful-space-efficiency">Heedful space-efficiency</h3> + +<p>A heedful contract is a set of classic higher-order contracts. When applying a new contract to a value, check whether the new contract is in the set. If so, ignore the new contract. If not, add the new contract to the set &mdash; or raise an error. Every value gets at most one set-wrapper, and each member of a set-wrapper represents a new constraint.</p> + +<p>To check a value against a set, for example when reading from a vector, check each of the elements in any order. If an element raises an error, report it.* Alternatively, an implementation can check all the elements and report all that disagree with the value.</p> + +<p>The heedful method is a compromise between forgetful and eidetic.</p> + +<ul> + <li> + <p>Unlike forgetful, heedful uses a new data structure to represent contacts and requires some kind of <code>contract-stronger?</code> predicate. Heedful also remembers (some of) the history of a value and catches the same errors as classic and eidetic contracts.</p></li> + <li> + <p>Unlike eidetic, heedful uses a simpler data structure with no need to keep duplicate flat contracts depending on the order they are encountered. Heedful cannot, however, uniquely identify the two parties involved in a contract error. In general, there are multiple contracts that a programmer must inspect to find the source of a mismatch.</p></li></ul> + +<p>For details, see <a href="https://arxiv.org/abs/1410.2813">the extended version</a> of Michael&rsquo;s POPL 2015 paper. Don&rsquo;t bother searching <a href="http://www.cs.pomona.edu/~michael/papers/popl2015_space.pdf">the conference version</a> &mdash; aside from one remark in Appendix B, heedful and forgetful are nowhere to be found.</p> + +<p><code>*</code> If an implementation promises to report one mismatch, instead of all mismatches, then it does not need to keep the full set of contracts. Thanks to <a href="http://mballantyne.net/">Michael Ballantyne</a> for explaining this to me.</p> + +<h3 id="priorities-and-appearances">Priorities and Appearances</h3> + +<p>The extended version of <em>Space-Efficient Manifest Contracts</em> introduces the forgetful and heedful methods with extreme modesty. It&rsquo;s tempting to skip past them and focus on the eidetic method.</p> + +<blockquote> + <p>&ldquo;Since eidetic and classic contracts behave the same, why bother with forgetful and heedful? First and foremost, the calculi offer insights into the semantics of contracts: the soundness of forgetful depends on a certain philosophy of contracts; heedful relates to threesomes without blame [<a href="https://dl.acm.org/citation.cfm?doid=1706299.1706342">Siek and Wadler 2010</a>]. Second, we offer them as alternative points in the design space. Finally and perhaps cynically, they are strawmen&mdash;warm up exercises for eidetic.&rdquo; &mdash; <a href="https://arxiv.org/abs/1410.2813"><em>Space-Efficient Manifest Contracts</em></a></p><!-- Section 1, bottom of page 2--></blockquote> + +<p>And yet, at least two other research papers rely on these &ldquo;strawmen&rdquo; &mdash; or rather, the ideas behind the names.</p> + +<p><a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a>, at ICFP 2017, demonstrates one technique for adding two varieties of types to a gradual language. The semantics in the paper is forgetful; if a higher-order value crosses multiple type boundaries, the intermediate server obligations disappear.</p> + +<blockquote> + <p>&ldquo;if a lambda abstraction is preceded by multiple casts, then the rule erases all of them, except for the last one&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- page 21--></blockquote> + +<p>This forgetfulness was a deliberate choice. A classic semantics would satisfy the same type soundness theorem, but the authors picked forgetful for its simplicity and performance implications.</p> + +<blockquote> + <p>&ldquo;removing these casts preserves the soundness of the evaluation while reducing the number of them&rdquo;</p> + <p>&ldquo;while this choice makes the calculus simpler without hindering soundness, it yields a formalism unfit to finger culprits&rdquo; &mdash; <a href="https://dl.acm.org/citation.cfm?id=3110285"><em>Gradual Typing with Union and Intersection Types</em></a></p><!-- p.27--><!-- page 21--></blockquote> +<!-- The followup at POPL 2019 is not forgetful.--> +<!-- It's similar to eager coercions ... keep all types around and error--> +<!-- if there's a new type that doesn't match the old ones.--> +<!-- Also, that paper chooses not to let functions have intersection types,--> +<!-- which kind-of-avoids the questions ... but really the eagerness is key.--> + +<p><a href="https://dl.acm.org/citation.cfm?id=3009849"><em>Big Types in Little Runtime</em></a>, at POPL 2017, presents a gradual typing system that avoids the use of wrappers. Instead, their <em>transient</em> semantics rewrites typed code ahead of time to mimic the checks that forgetful contracts would perform. These checks suffice for a shallow type soundness theorem.</p> + +<p>That paper also introduces a heedful-like strategy for improving the error messages produced by a forgetful check. The strategy adds a global map to the semantics; keys in the map are unique identifiers for values (heap addresses), and values are sets of types. When a value meets a compatible type, the type is added to the value&rsquo;s set. When a mismatch occurs, the semantics <a href="https://www.ccs.neu.edu/home/types/resources/notes/transient-undefined-blame-extract.pdf">tries to report</a> every type in the set that relates to the mismatch.</p> + +<p>And so, forgetful and heedful were edged out of POPL 2015 but managed to sneak in to POPL 2017. Since then, forgetful appeared in ICFP 2017 and, briefly, in <a href="https://www2.ccs.neu.edu/racket/pubs/icfp18-gf.pdf">ICFP 2018</a>. Where will we see them next?</p> + + The Behavior of Gradual Types: A User Study + http://prl.ccs.neu.edu/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/?utm_source=migratory-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-11-the-behavior-of-gradual-types-a-user-study + Tue, 11 Dec 2018 19:50:33 UT + Ben Greenman + <!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>The Behavior of Gradual Types: A User Study</em> by Preston Tunnell&mdash;Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi. For the full paper, datasets, and slides, <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#tgpk-dls-2018">click here</a>.</p></blockquote> + +<p>The long-term goal of gradual typing is to build languages that offer the &ldquo;best&rdquo; of both static and dynamic typing. Researchers disagree, however, on what the semantics of a mixed-typed language should be; there are <a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">at least three competing proposals</a> for combining a dynamically-typed language with a similar statically-typed language.</p> + +<blockquote> + <p>It&rsquo;s an interesting situation. There are dozens of papers on the semantics of gradual types&mdash;and <a href="http://www.ccs.neu.edu/home/types/resources/talks/tgpk-dls-2018.pdf">many claim</a> to have developers in mind&mdash;but zero papers that ask developers what they think.</p></blockquote> + +<p>To help inform the discussion, we recently designed a <a href="http://cs.brown.edu/research/plt/dl/dls2018">survey</a> to see what programmers think of three mixed-typed semantics. The survey is based on 8 example programs; we selected these 8 programs because the set as a whole tells the three mixed-typed semantics apart. For each program, the survey presents a few possible outcomes of running the program and asks participants for their opinion on each outcome.</p> + +<p>The image below shows one program from the survey:</p> + +<p> <img src="/img/gtsurvey-example-program.png" alt="Figure 1: example program" /></p> + +<p>This program creates an array, passes it between typed and untyped variables, and performs write &amp; read operations. What should happen when we run this program? One option is to ignore the type annotations and return the second element of the array (<code>"bye"</code>). A second option is to reject the write operation (on line 4) because it attempts to write a number to a variable of type <code>Array(String)</code>. A third option is to reject the assignment after the read operation (on line 5) because it attempts to assign a string to a variable of type <code>Number</code>. These are the three behaviors in the survey:</p> + +<p> <img src="/img/gtsurvey-example-behaviors.png" alt="Figure 2: behaviors for the example question" /></p> + +<blockquote> + <p>A fourth option is to reject the assignment of an <code>Array(String)</code> to a variable of type <code>Array(Number)</code>. A few participants left comments asking for this behavior. See the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for their comments, and see <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for why we left that behavior out.</p></blockquote> + +<p>For each behavior, we asked for respondents&rsquo; preference along two independent dimensions:</p> + +<ul> + <li>Do you <em>like</em> or <em>dislike</em> this behavior?</li> + <li>Does it match your <em>expectation</em> as a programmer?</li></ul> + +<p>Combined, the dimensions lead to four possible <em>attitudes</em>: Like and Expected, Like and Unexpected, Dislike and Expected, Dislike and Unexpected. The full example question, with attitudes and space for comments, is below.</p> + +<p> <img src="/img/gtsurvey-example-question.png" alt="Figure 3: complete question" /></p> + +<p>We administered the survey to three populations &mdash; software engineers, students, and Mechanical Turk workers &mdash; and thereby collected three sets of attitudes for each question. The results for the running example are below:</p> + +<p> <img src="/img/gtsurvey-example-data.png" alt="Figure 4: results for Question 7" /></p> + +<p>The figure is a matrix of three columns (one for each population) and three rows (one for each behavior). Each cell of the matrix contains a bar chart showing the attitudes that we collected.</p> + +<blockquote> + <p>Unlike the survey question, the behaviors in the results are labeled as <strong>Deep</strong>, <strong>Erasure</strong>, and <strong>Shallow</strong>. These names describe the three mixed-typed semantics.</p></blockquote> + +<p>For this question, the software engineers (left column, green bars) mostly picked the &ldquo;Dislike and Unexpected&rdquo; attitude for every behavior. The students (mid column, blue bars) also show consensus on &ldquo;Dislike and Unexpected&rdquo; for the <strong>Deep</strong> and <strong>Erasure</strong> behaviors; however, they are split for the <strong>Shallow</strong> behavior. The Mechanical Turk workers are divided on every behavior.</p> + +<p>See <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for the other questions and responses.</p> + +<p>Overall, our main finding is that respondents preferred behaviors that enforced full types and reported runtime mismatches as early as possible. The takeaway is thus:</p> + +<p style="margin-left: 40px; margin-right: 40px">if you are designing a mixed-typed language and choose <strong>not</strong> to enforce full types, then make sure to explain this behavior to users!</p> + +<p>Put lots of example programs in the language&rsquo;s documentation. The programs in the survey can be adapted to explain how your chosen behavior differs from alternatives.</p> + +<h2 id="questions">Questions</h2> + +<p>Here are some good questions we&rsquo;ve gotten that are not clearly answered in the paper.</p> + +<h4 id="q-did-any-respondents-expect-more-than-one-behavior">Q. Did any respondents &ldquo;expect&rdquo; more than one behavior?</h4> + +<p>Yes, 59% <!-- 20/34--> of the software engineers and 82% <!-- 14/17--> of the students selected &ldquo;Liked and Expected&rdquo; and/or &ldquo;Dislike and Expected&rdquo; for different behaviors on the same program.</p> +<!-- They probably interpreted "Expected" as--> +<!-- "the program does something that makes sense", rather than--> +<!-- "the program does the one thing that I believe it should do".--> +<!-- ids for "double-expect" S.Es : R_24bz47lgcAOkCux R_2R4dZ1l0t3yx6fW R_b7yMVe7VtmmsrHb R_31MXSUfCyDE8FdG R_6LGXyOirYNtYWd3 R_2qyMZBAs74PrsSz R_2ASFRBh2jfuRgP1 R_1PUc0AUEzdXKGt8 R_2dL60N9oPIkbvWY R_1BXXqYyxH7R4r9l R_1ON2sxGalcODyAd R_1oyZasBudU5gKPS R_1FIHgkQbWGaxuHd R_b1s2YMBWCrCRvxf R_29t0zWxkQsfb9FT R_2fevZOrFGzS6JLf R_8Dn6NMjDyigT59n R_2pRG370z3cBUaKv R_2qDXTFI53ntWMu4 R_ZI8AwATueqyWwOR--> +<!-- ids for "double-expect" students : R_9B6WHWEX5l0DskN R_22VAu37cGWQPQx1 R_3hgYSaGy2tbyY3G R_3rTbAqgn1rhQK4d R_r3HqAP1yGRXHaZX R_1l05qvQ1sYOCcCF R_3qaMT9xR7CRYg2Y R_1Li0sGHkxk1VfcA R_24ITtgvBzg9RpE3 R_3HzshHbDWkayp4t R_5mtEFLtSX0iPVOp R_1IR6vdpmVw4OCqV R_2XpWlkKjH9LQqln R_DoQrROe0dcb1YJz--> + +<h4 id="q-did-the-respondents-have-a-prior-preference-for-static-or-dynamic-typing">Q. Did the respondents have a prior preference for static or dynamic typing?</h4> + +<p>Near the end of the survey we asked: &ldquo;Which do you prefer, typed or untyped programming?&rdquo;. See table 2 of <a href="http://cs.brown.edu/~sk/Publications/Papers/Published/tgpk-beh-grad-types-user-study">the paper</a> for coded responses to this question, or the <a href="http://cs.brown.edu/research/plt/dl/dls2018">anonymized responses</a> for the ground truth. Most preferred typed programming.</p> + + Java and Migratory Typing + http://prl.ccs.neu.edu/blog/2018/12/02/java-and-migratory-typing/?utm_source=migratory-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + Sun, 02 Dec 2018 14:41:53 UT + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> + + A Spectrum of Type Soundness and Performance + http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/?utm_source=migratory-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-06-a-spectrum-of-type-soundness-and-performance + Sat, 06 Oct 2018 11:23:35 UT + Ben Greenman + +<p>The literature on mixed-typed languages presents (at least) three fundamentally different ways of thinking about the integrity of programs that combine statically typed and dynamically typed code. Recently, we have been sorting them out.</p> +<!-- more--> + +<blockquote> + <p>Note: this post is an extended abstract for the paper <em>A Spectrum of Type Soundness and Performance</em> by Ben Greenman and Matthias Felleisen. For the full paper, slides, code, and a video presentation, visit <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018">http://www.ccs.neu.edu/home/types/publications/publications.html#gf-icfp-2018</a></p></blockquote> + +<p>A dynamically-typed language runs any program that &ldquo;looks good&rdquo; (i.e., passes some basic syntactic criteria. In Python a program cannot mix indentation levels. In Racket a program cannot refer to unbound variables). A statically-typed language runs any program that both &ldquo;looks good&rdquo; and is well-typed according to a type checker.</p> + +<p>A <em>mixed-typed</em> language allows some combination of static and dynamic typing. There are many languages that fall in the mixed-typed category; figure 1 lists a few, roughly arranged left-to-right by the year they first provided a way to mix.</p> + +<div class="figure"><img src="/img/mixed-typed-systems-by-year.png" alt="Figure 1: Some mixed-typed languages" /> + <p class="caption">Figure 1: Some mixed-typed languages</p></div> + +<p>These languages all try to combine static and dynamic typing in a useful way, but they take VERY different approaches. For example:</p> + +<ul> + <li><strong>MACLISP</strong> defines a syntax for type annotations but does not say how a compiler should interpret the types; see section 14.2 of the <a href="http://www.softwarepreservation.org/projects/LISP/MIT/Moon-MACLISP_Reference_Manual-Apr_08_1974.pdf">Moonual</a>. For example, a compiler may use types to generate specialized code that assumes the type annotations are correct (and has undefined behavior otherwise).</li> + <li><strong>Strongtalk</strong> includes a static type checker and DOES NOT use types to change the behavior of a program. For rationale, see the <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a> position paper.</li> + <li><strong>Typed Racket</strong> lets a program combine statically-typed modules and dynamically-typed modules. The compiler inserts run-time checks at the boundaries between such modules to detect any mismatches between the static types and incoming dynamically-typed values.</li> + <li><strong>Thorn</strong> requires that every value in a program has a type, but allows dynamically-typed contexts to manipulate values. In other words, every Thorn value is an instance of a statically-declared class and classes may contain dynamically-typed methods.</li> + <li><strong>Reticulated</strong> lets a program combine static and dynamic <em>expressions</em> and guarantees that the combination has a well-defined semantics (Vitousek, Swords, and Siek <a href="https://dl.acm.org/citation.cfm?id=3009849">POPL 2017</a>).</li></ul> + +<p>That makes five different systems. There are 15 other systems in the figure, and many more in the world. How can we make sense of this space? We claim: by understanding each system&rsquo;s protocol for checking dynamically-typed values at a <em>type boundary</em> (between static and dynamic code).</p> + +<h3 id="main-contribution">Main Contribution</h3> + +<p>In the paper <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>A Spectrum of Type Soundness and Performance</em></a>, we define a tiny mixed-typed language and show three ways to define the behavior of programs &mdash; based on three protocols for checking dynamically-typed values that cross a boundary into statically-typed code.</p> + +<p>The three behaviors are inspired by existing languages. A <strong>higher order</strong> behavior ensures that dynamically-typed values match the static type at a boundary &mdash; by checking the value when possible, and by monitoring the value&rsquo;s future interactions when necessary. A <strong>first order</strong> behavior performs a yes-or-no check on dynamically-typed values and never monitors their future behavior. An <strong>erasure</strong> behavior does no checking whatsoever.</p> + +<blockquote> + <p>Example (monitors): if typed code expects a function from numbers to numbers and receives an untyped function <code>f</code>, then one way to enforce the type boundary is to wrap <code>f</code> in a proxy to assert that every future call to <code>f</code> returns a number. In this case, the proxy monitors the behavior of <code>f</code>.</p></blockquote> + +<p>Concretely, the paper defines three formal semantics with the same names. The <strong>higher-order</strong> semantics enforces full types at the boundaries (Section 2.3). The <strong>first-order</strong> semantics enforces type constructors at the boundaries, and furthermore enforces type constructors on every &ldquo;selector&rdquo; operation in typed code, e.g., function application, data structure access (Section 2.5). The <strong>erasure</strong> semantics simply ignores the types (Section 2.4).</p> + +<p>Each semantics satisfies a <em>different</em> notion of soundness for mixed-typed programs, and each notion is slightly weaker than soundness for fully-typed programs. The paper states these theorems (Section 2) and the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">online supplement</a> gives full proofs.</p> + +<p>The paper has more to say about the meta-theory. See section 2 and section 4.</p> + +<blockquote> + <p>To the best of our knowledge, this paper is the first to explicitly acknowledge that different approaches to a mixed-typed language lead to different notions of soundness. Other papers state type soundness theorems for <a href="https://dl.acm.org/citation.cfm?id=2676971">subset of the language</a> (in the spirit of <a href="http://soundiness.org/">soundiness</a>) or use the name &ldquo;type soundness&rdquo; to describe <a href="https://dl.acm.org/citation.cfm?id=2676971">a different property</a>.</p></blockquote> + +<p>Next, we used the three semantics as a guide to arrive at three compilers for Typed Racket. The higher-order compiler is the standard Typed Racket. The first-order compiler is something we built, based on the semantics. The erasure compiler simply ignores the type annotations &mdash; similar to Typed Racket&rsquo;s <a href="http://docs.racket-lang.org/ts-reference/Typed_Racket_Syntax_Without_Type_Checking.html">no-check</a> language.</p> + +<p>Using this set-up, we measured the performance of mixed-typed programs via each compiler using the method suggested by Takikawa et. al (<a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">POPL 2016</a>). The programs we measured are the non-object-oriented ones from our <a href="http://docs.racket-lang.org/gtp-benchmarks/index.html">benchmark suite</a>.</p> + +<p>To some extent, the performance results confirm conjectures from the literature. The full results, however, include many surprises &mdash; see section 3 of the paper, section B of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, and/or the <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slides</a>.</p> + +<h3 id="implications">Implications</h3> + +<ol> + <li>The model in the paper is one way to understand the different approaches to mixed-typed languages. See section 5 of the paper, section D of the <a href="https://repository.library.northeastern.edu/files/neu:cj82rk279">supplement</a>, or <a href="http://www.ccs.neu.edu/home/types/publications/apples-to-apples/gf-icfp-2018-slides.pdf">slide 114</a>.</li> + <li>Programmers using mixed-typed languages need to know what guarantees their types provide. (It is <a href="https://twitter.com/jbandi/status/965005464638541825">not safe to assume that TypeScript types give the same guarantees as OCaml types</a>!) Section 4 of the paper contains many examples of how the different guarantees may affect practice.</li> + <li>The relative performance of different approaches is more nuanced than the literature suggests. Our paper gives a first systematic comparison based on implementations that have clear areas for improvement. The question is: can we find improvements that lead to asymptotic differences, or is it a battle for constant factors?</li></ol> + +<blockquote> + <p>Note: in this post, a <em>mixed-typed language</em> is one that allows any combination of static and dynamic typing. A <em>gradually-typed language</em> is one that allows a certain kind of mixing that satisfies properties defined by Siek, Vitousek, Cimini, and Boyland (<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>).</p></blockquote> + + Sampling Gradual Typing Performance + http://prl.ccs.neu.edu/blog/2018/05/08/sampling-gradual-typing-performance/?utm_source=migratory-typing&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + Tue, 08 May 2018 15:37:37 UT + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> \ No newline at end of file diff --git a/blog/feeds/monotonicity.atom.xml b/blog/feeds/monotonicity.atom.xml new file mode 100644 index 00000000..079c4be0 --- /dev/null +++ b/blog/feeds/monotonicity.atom.xml @@ -0,0 +1,144 @@ + + + PRL Blog: Posts tagged 'monotonicity' + + + urn:http-prl-ccs-neu-edu:-blog-tags-monotonicity-html + 2017-10-22T11:59:06Z + + Monotonicity Types: Towards A Type System for Eventual Consistency + + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + 2017-10-22T11:59:06Z + 2017-10-22T11:59:06Z + + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> \ No newline at end of file diff --git a/blog/feeds/monotonicity.rss.xml b/blog/feeds/monotonicity.rss.xml new file mode 100644 index 00000000..68d0c6d5 --- /dev/null +++ b/blog/feeds/monotonicity.rss.xml @@ -0,0 +1,144 @@ + + + + PRL Blog: Posts tagged 'monotonicity' + PRL Blog: Posts tagged 'monotonicity' + http://prl.ccs.neu.edu/blog/tags/monotonicity.html + Sun, 22 Oct 2017 11:59:06 UT + Sun, 22 Oct 2017 11:59:06 UT + 1800 + + Monotonicity Types: Towards A Type System for Eventual Consistency + http://prl.ccs.neu.edu/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/?utm_source=monotonicity&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + Sun, 22 Oct 2017 11:59:06 UT + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> \ No newline at end of file diff --git a/blog/feeds/observational-equivalence.atom.xml b/blog/feeds/observational-equivalence.atom.xml new file mode 100644 index 00000000..5f067ba2 --- /dev/null +++ b/blog/feeds/observational-equivalence.atom.xml @@ -0,0 +1,285 @@ + + + PRL Blog: Posts tagged 'observational equivalence' + + + urn:http-prl-ccs-neu-edu:-blog-tags-observational-equivalence-html + 2017-09-27T15:44:57Z + + Final Algebra Semantics is Observational Equivalence + + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + 2017-09-27T15:44:57Z + 2017-09-27T15:44:57Z + + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> \ No newline at end of file diff --git a/blog/feeds/observational-equivalence.rss.xml b/blog/feeds/observational-equivalence.rss.xml new file mode 100644 index 00000000..d4ac67b4 --- /dev/null +++ b/blog/feeds/observational-equivalence.rss.xml @@ -0,0 +1,285 @@ + + + + PRL Blog: Posts tagged 'observational equivalence' + PRL Blog: Posts tagged 'observational equivalence' + http://prl.ccs.neu.edu/blog/tags/observational-equivalence.html + Wed, 27 Sep 2017 15:44:57 UT + Wed, 27 Sep 2017 15:44:57 UT + 1800 + + Final Algebra Semantics is Observational Equivalence + http://prl.ccs.neu.edu/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/?utm_source=observational-equivalence&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-27-final-algebra-semantics-is-observational-equivalence + Wed, 27 Sep 2017 15:44:57 UT + Max New + +<p>Recently, &ldquo;final encodings&rdquo; and &ldquo;finally tagless style&rdquo; have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, <a href="https://github.com/michaelballantyne">Michael Ballantyne</a>, <a href="http://ccs.neu.edu/home/ryanc">Ryan Culpepper</a> and I asked &ldquo;in what category are these actually final objects&rdquo;? As it turns out our very own <a href="http://www.ccs.neu.edu/home/wand/">Mitch Wand</a> wrote one of the first papers to make exactly this idea precise, so I read it <a href="https://www.cs.indiana.edu/ftp/techreports/TR65.pdf">available here</a> and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.</p> + +<p>In this post, I&rsquo;ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we&rsquo;ll learn a bit about categorical logic, and I&rsquo;ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.</p> +<!-- more--> + +<h1 id="intuition-implementing-a-signature">Intuition: Implementing a Signature</h1> + +<p>As a running example, say we wanted to implement a datatype of finite maps whose keys and values are both integers, i.e., finite multisets of integers.</p> + +<p>We could specify such a datatype by specifying a little language of numbers and finite multisets. We&rsquo;ll have two &ldquo;sorts&rdquo; <code>num</code> and <code>multiset</code>, a constant for every integer, and an addition function</p> + +<pre><code>'n : () -&gt; num; +add : (num, num) -&gt; num</code></pre> + +<p>subject to the silly-looking equation:</p> + +<pre><code>add('n,'m) = '(n + m)</code></pre> + +<p>and some operations on multisets</p> + +<pre><code>empty : () -&gt; multiset; +singleton : (num) -&gt; multiset; +union : (multiset, multiset) -&gt; multiset; +remove : (num, multiset) -&gt; multiset; +count : (num, multiset) -&gt; num</code></pre> + +<p>subject to the computational equations:</p> + +<pre><code>count('n, empty) = '0 +count('n, singleton('n)) = '1 +count('n, singleton('m)) = '0 +count('n, union(s,t)) = add(count('n,s), count('n, t)) +count('n, remove('n,s)) = '0 +count('n, remove('m,s)) = count('n,s)</code></pre> + +<p>These are &ldquo;all&rdquo; of the equations we need to actually run our programs and get a number out, but not all the equations we intuitively <em>want</em> for reasoning about our programs. For instance, clearly <code>union</code> should be commutative, and <code>remove</code> should be idempotent, but it&rsquo;s impossible to prove that with just the equations specified. In fact, we can make a model of this theory that refutes them by constructing the &ldquo;initial algebra&rdquo;. In Haskell, we could say</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Union</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Remove</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"></span> +<span class="w"> </span><span class="kr">deriving</span><span class="w"> </span><span class="p">(</span><span class="kt">Eq</span><span class="p">)</span><span class="w"></span> + +<span class="nf">count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="kt">Empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="n">m</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="nf">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">Remove</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">count</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then it is completely obvious that all of our equations hold, but then <code>Union</code> is <em>not</em> commutative, as ghci will tell us:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="p">(</span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="kt">Union</span><span class="p">`</span><span class="w"> </span><span class="kt">Singleton</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span> +<span class="kt">False</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, there is another encoding that will give us that <code>union</code> is commutative and <code>remove n</code> is idempotent and actually every equation we could possibly want! It&rsquo;s called the &ldquo;final encoding&rdquo; or &ldquo;final algebra&rdquo;. In Haskell, this looks like:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">count&#39;</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span> +<span class="nf">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="n">m</span><span class="w"> </span><span class="n">n</span><span class="w"></span> + +<span class="nf">empty</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">empty</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">singleton</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">singleton</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">union</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">remove</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"></span> +<span class="nf">remove</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">m</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">m</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="mi">0</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="p">}</span><span class="w"></span> + +<span class="nf">test&#39;</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="nf">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="nf">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we can verify that <code>union</code> is commutative because</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="nf">union</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">MultiSet&#39;</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">_count</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="nf">\</span><span class="n">n</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w"></span> +<span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">union</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="n">s</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>since <code>+</code> is commutative. Equality isn&rsquo;t decidable anymore so I can&rsquo;t give you a simple piece of code to witness this, but we can test our example before and we won&rsquo;t be able to distinguish them, no surprise:</p> + +<div class="brush: Haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">`</span><span class="n">union</span><span class="p">`</span><span class="w"> </span><span class="n">singleton</span><span class="w"> </span><span class="mi">1</span><span class="w"></span> +<span class="o">&gt;</span><span class="w"> </span><span class="n">and</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">count&#39;</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">&lt;-</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">1000</span><span class="p">]]</span><span class="w"></span> +<span class="kt">True</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>How do we know this is the &ldquo;best&rdquo; or at least &ldquo;most canonical&rdquo; implementation of our datatype? The intuition is that we really don&rsquo;t care at all <em>how</em> our multisets are implemented as long as they behave the right way with respect to <code>count</code> since <code>count</code> returns an <code>Integer</code>, a type we do understand. Our encoding accomplishes this by representing a multiset <code>s</code> by the partially applied function <code>\n -&gt; count n s</code>.</p> + +<p>The formal name for this idea is <em>observational equivalence</em>. We say that two closed terms <code>s,t</code> of sort <code>multiset</code> are <em>observationally equivalent</em> if for any term <code>C</code> of type <code>num</code> that has <code>s</code> as a subterm, we can swap <code>t</code> in for <code>s</code> and prove that the two terms are equal. For instance <code>C</code> might be <code>count(3, union(s, singleton(3)))</code> or <code>add(4,remove(5,s))</code>. Then we&rsquo;ve reduced the possibly complicated equality for <code>multiset</code> to the simple equality of <code>num</code>.</p> + +<p>Proving that the final encoding above satisfies all observational equivalences is beyond the scope of this blog post (see <a href="https://hal.inria.fr/inria-00076514/document">here</a>), but let&rsquo;s see what all this talk about &ldquo;algebras&rdquo;, initial or final is all about.</p> + +<h1 id="formalization-attempt-1-algebras-of-a-theory">Formalization Attempt 1: Algebras of a Theory</h1> + +<p>First, our little language of numbers and multisets is called a <em>theory</em>. The specific category gadget that we&rsquo;ll use to describe it is a <em>multi-sorted Lawvere theory</em>, or just <em>Lawvere theory</em> for short.</p> + +<p>A <em>Lawvere theory</em> is a category with finite products all of whose objects are finite products of a collection of <em>sorts</em> \(S\). We can construct this category from our little language above by making the objects be <em>contexts</em> \(x:num,y:multiset,...\) and morphisms \(\Gamma \to +x_1:s_1,...,x_n:s_n\) to be \(n\)-tuples of terms \(\Gamma \vdash t_1 : s_1,..., +\Gamma \vdash t_n : s_n\) <em>modulo</em> the equations we&rsquo;ve specified. We&rsquo;ll use the letter \(T\) to mean a Lawvere theory.</p> + +<p>Then a <em>\(T\)-algebra</em> is a denotational semantics of our theory \(T\), i.e., a product preserving functor \(A : T \to Set\). This means for every sort we get a set \(A(s)\) and for every term \(x_1:s_1,...,x_n:s_n +\vdash t : s\) a function \(A(t) : A(s_1)\times\cdots \times A(s_n) \to +A(s)\).</p> + +<p>Finally a <em>morphism of \(T\)-algebras</em> from \(A\) to \(B\) is a way to translate one algebra into another. Briefly, it is a natural transformation from \(A\) to \(B\), but concretely this means for every sort \(s\) we get a function \(\alpha_s : A(s) \to B(s)\) that translates \(A\)s interpretation of \(s\) as a set into \(B\)s. The key property that we want is that the operations according to \(A\) and \(B\) do the same thing as determined by \(\alpha\). Specifically, for any term \(x_1:s_1,...,x_n:s_n \vdash t : +s\), and inputs \(x_1 \in A(s_1),...,x_n \in A(s_n)\) we should get the same result if we evaluate \(A(t)(x_1,\ldots,x_n)\) and then apply \(\alpha_s\) as if we first translate \(x_1,\ldots,x_n\) to \(B(s_1),\ldots,B(s_n)\) and then apply \(B(t)\). If you unwind the definitions, this is exactly what naturality says.</p> + +<p>Then we have a category we&rsquo;ll call \(T-Alg\) of \(T\)-algebras and we can ask if there are initial or final algebra. It turns out that both of them <em>always</em> exist.</p> + +<p>The initial algebra is most famous here, we define for each sort \(In(T)(s) = \cdot \vdash s\), the closed terms of that sort modulo the equivalence of the theory, and \(In(T)(s_1,\ldots,s_n) = +In(T)(s_1)\times\ldots,In(T)(s_n)\). Then the terms are just interpreted as the functions you get by plugging closed inputs into them. Then if we look at what what a morphism of \(T\)-algebras from \(In(T) \to A\) is, we see that we don&rsquo;t have any choice, the only one is the one that maps \(\cdot \vdash t : s\) to \(A(t)\) and this makes all the right diagrams to commute. This is pretty similar to our definition of &ldquo;initial algebra&rdquo; before, except that this time we defined <code>count</code> as a function, not just a case of an ADT, but that was just an easy way to satisfy the computational equations for <code>count</code>.</p> + +<p>However, an egregious flaw presents itself when we look at what the <em>final</em> algebra is. It&rsquo;s completely trivial! We can define \(Fin(T)\) to take every sort to a one element set \(Fin(T)(s) = \{*\}\) and every term to the trivial function \(\{*\}^n \to \{*\}\). What the hell? This interprets numbers and multisets as trivial one-element sets. To rule this one out, we need to add some conditions to our algebras.</p> + +<h1 id="formalization-algebras-of-a-theory-extension">Formalization: Algebras of a Theory Extension</h1> + +<p>To rule out these boring algebras, and get a nice final algebra, we have to recognize that the sorts <code>num</code> and <code>multiset</code> in our theory are not really on equal footing. While we are not sure how multisets should be defined, we know <em>exactly</em> what numbers are!</p> + +<p>To formalize this we&rsquo;ll call the full theory \(T_1\) and the theory with just numbers \(T_0\). Then there should be a map from \(T_0\) to \(T_1\) that is the inclusion of theories. We&rsquo;ll formalize this as a <em>morphism of theories</em>. A morphism of theories is a <em>strict</em> product-preserving functor from one theory to another. The strictness ensures that we don&rsquo;t mix up our sorts and our contexts, a morphim of theories has to map sorts to sorts, whereas a non-strict functor could map a sort to a context with two sorts it&rsquo;s equivalent to. What this really amounts to is a translation of one theory into another. It maps sorts to sorts and terms to terms of the appropriate sorts in a compositional way. However, we don&rsquo;t want to consider <em>all</em> such morphisms, only the ones that are &ldquo;conservative extensions&rdquo;, which means</p> + +<ol> + <li>there are no new closed terms at old types</li> + <li>closed terms that were different before remain different.</li></ol> + +<p>In our example (1) ensures that we don&rsquo;t add any new exotic numbers like <code>undefined</code> or <code>∞</code>, and (2) ensures that we keep \(0\) different from \(1\), like the final algebra did before by having all numbers have the same interpreation \(*\).</p> + +<p>We can formalize this in the following way. Note that any morphism of Lawvere theories \(m : T \to S\) induces a <em>functor</em> on the category of algebras \(m^* : S-Alg \to T-Alg\) by just composing functors. An \(S\)-algebra is a functor from \(S\) to sets, and \(m\) is a functor from \(T\) to \(S\) so we can compose to get \(m^*(A)(t) = A(m(t))\).</p> + +<p>Now, we can express the idea of a conservative extension by saying that the canonical arrow from \(In(T)\) to \(m^*(In(S))\) is an isomorphism. Recalling the definition of initial algebras, this says exactly that the closed terms in \(T\) up to \(T\)-equivalence are isomorphic to the closed terms of the type provided by \(m\) in \(S\) up to \(S\)-equivalence. This is an equivalent formulation to the definition in Mitch&rsquo;s paper, but there it is separated into two properties fullness and faithfulness, and doesn&rsquo;t use the initial algebras and \(m^*\) explicitly.</p> + +<p>Now we can verify that the inclusion \(i : T_0 \to T_1\) of the number theory into the number-multiset theory is an extension in this sense.</p> + +<p>Finally we can define our notion of \(i\)-algebra, which will be our correct notion of algebra. An \(i\)-algebra is a \(T_1\) algebra \(A\) such that</p> + +<ol> + <li>The canonical algebra map \(! : In(T_0) \to m^*A\) is an isomorphism.</li> + <li>The canonical algebra map \(! : In(T_1) \to A\) is surjective i.e., for each sort \(s, !_s\) is surjective.</li></ol> + +<p>The first condition says again that we have a conservative extension of \(T_0\), but the second is more interesting. It says that every denotation given by \(A\) is represented by some term in \(T_1\). In fact what it really ensures is that \(A\) determines a <em>congruence relation</em> on \(T_1\) given by \(t1 \equiv_A t2\) if \(A(t1) = A(t2)\). In light of this, the first condition could be called <em>adequacy</em>.</p> + +<p>Furthermore, the surjectivity condition ensures that any morphism of \(i\) algebras, i.e., a map as \(T_1\)-algebras is also surjective, so a morphism \(A \to B\) is a witness to the fact that \(B\) determines a <em>stronger</em> congruence relation on \(T_1\) than \(A\) does: \(t1 \equiv_B t2 +\implies t1 \equiv_A t2\). Then asking for a final algebra is asking for exactly the:</p> + +<blockquote> + <p>Strongest adequate congruence relation</p></blockquote> + +<p>which is exactly the definition of observational equivalence you will find in, say Pitt&rsquo;s chapter of <a href="https://www.cis.upenn.edu/~bcpierce/attapl/">Advanced TAPL</a>. There is a difference in the meaning of <em>adequacy</em>, though. Usually adequacy is defined in terms of an operational semantics, but here everything is based on an axiomatic notion of equality, but I think they play the same role in the two settings, so I think it&rsquo;s reasonable to use the same word. On thing I like about this formulation is very nice though since it makes obvious that <em>adequacy</em> is not a predetermined concept, we have to pick \(T_0\) and \(i\) in order to know what adequacy means.</p> + +<h1 id="conclusion-tying-it-back-to-final-encodings">Conclusion: Tying it back to Final Encodings</h1> + +<p>So now we&rsquo;ve seen that</p> + +<blockquote> + <p>Final algebras are equivalent to initial algebras modulo observational equivalence</p></blockquote> + +<p>Of course we haven&rsquo;t precisely gotten back to where we started: we were talking about denotational semantics in terms of sets and functions, but what we really want are implementations in our favorite programming languages. Fortunately, we didn&rsquo;t use very many properties of sets in our definition, so it&rsquo;s pretty easy to swap out the category of Sets for some category built out of the terms of our programming language. We can also swap out sets for some much cooler category of denotations like domains or metric spaces or time-varying values.</p> + +<p>Another question is how to implement this when we have a proper <em>type theory</em> and not just some boring sorts. In particular, if we have function types, then we won&rsquo;t be able to get functions from functions in our term model to functions in our denotations due to contravariance. Perhaps logical relations are the solution?</p> \ No newline at end of file diff --git a/blog/feeds/ocaml.atom.xml b/blog/feeds/ocaml.atom.xml new file mode 100644 index 00000000..b5c3d4ac --- /dev/null +++ b/blog/feeds/ocaml.atom.xml @@ -0,0 +1,456 @@ + + + PRL Blog: Posts tagged 'ocaml' + + + urn:http-prl-ccs-neu-edu:-blog-tags-ocaml-html + 2016-05-24T10:51:34Z + + Measuring GC latencies in Haskell, OCaml, Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + 2016-05-24T10:51:34Z + 2016-05-24T10:51:34Z + + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/ocaml.rss.xml b/blog/feeds/ocaml.rss.xml new file mode 100644 index 00000000..e50249f4 --- /dev/null +++ b/blog/feeds/ocaml.rss.xml @@ -0,0 +1,456 @@ + + + + PRL Blog: Posts tagged 'ocaml' + PRL Blog: Posts tagged 'ocaml' + http://prl.ccs.neu.edu/blog/tags/ocaml.html + Tue, 24 May 2016 10:51:34 UT + Tue, 24 May 2016 10:51:34 UT + 1800 + + Measuring GC latencies in Haskell, OCaml, Racket + http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/?utm_source=ocaml&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + Tue, 24 May 2016 10:51:34 UT + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/offsite.atom.xml b/blog/feeds/offsite.atom.xml new file mode 100644 index 00000000..4ff4c21b --- /dev/null +++ b/blog/feeds/offsite.atom.xml @@ -0,0 +1,550 @@ + + + PRL Blog: Posts tagged 'offsite' + + + urn:http-prl-ccs-neu-edu:-blog-tags-offsite-html + 2020-01-15T12:16:35Z + + The Typed Racket Optimizer vs. Transient + + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + 2020-01-15T12:16:35Z + 2020-01-15T12:16:35Z + + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> + + PRL Offsite 2019 Retrospective + + urn:http-prl-ccs-neu-edu:-blog-2019-12-12-prl-offsite-2019-retrospective + 2019-12-12T12:51:53Z + 2019-12-12T12:51:53Z + Ben Greenman + Olek Gierczak + + Ben Greenman, Olek Gierczak + +<p>On November 11th 2019, the PRL had a private offsite meeting at the <a href="https://mtwyouth.org">More Than Words bookstore</a> in downtown Boston. For future offsite organizers, this post records what happened and how.</p> +<!-- more--> + +<h2 id="early-planning-goals">Early Planning, Goals</h2> + +<p>Every Fall, the PRL holds a kickoff meeting to assign <a href="http://prl.ccs.neu.edu/contact">roles</a> for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.</p> + +<blockquote> + <p>Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the <a href="http://www.warrencenter.com">Warren Conference Center</a> is <a href="https://www.northeastern.edu/events/northeastern-owned-off-campus-venues">Northeastern University&rsquo;s only franchised</a> off-campus venue.</p></blockquote> + +<p>The <strong>motivation</strong> for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students &mdash; both to get to know the group, and to become known. They&rsquo;re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">Spring 2017 edition</a> of HOPL.</p> + +<p>Our primary <strong>goal</strong> this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.</p> + +<p>In light of the main goal, we prefer to call the event an &ldquo;offsite&rdquo; (or &ldquo;offsite meeting&rdquo;) rather than a &ldquo;retreat&rdquo; because the target was an informative day rather than a relaxing one.</p> + +<h2 id="booking-and-logistics">Booking and Logistics</h2> + +<p>Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that&rsquo;s all we had.</p> + +<h3 id="date--nov-11">Date = Nov 11</h3> + +<p>In the beginning, we chose Veterans&rsquo; Day (2019&ndash;11&ndash;11) and Northeastern <a href="https://registrar.northeastern.edu/app/uploads/2019-2020-University-Wide-Calendar-List.pdf">reading day</a> (2019&ndash;12&ndash;05) as possible dates. We ended up with Veterans&rsquo; Day.</p> + +<p>A small number of lab members were opposed to Veterans&rsquo; Day. They gave two reasons: the Fall semester is especially busy, and Veterans&rsquo; Day is a federal holiday.</p> + +<h3 id="venue--mtw">Venue = MTW</h3> + +<p>The first venue we tried was the <a href="http://www.warrencenter.com">Warren Conference Center</a> in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the <a href="https://www.khoury.northeastern.edu/people/chelsea-smith/">Khoury Events Team</a> to reach out on our behalf. They returned with an email address and <a href="/blog/static/warren-center-meetings-and-retreats.pdf">event menu</a> (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.</p> +<!-- Donald Pepple (Northeastern) made contact with Christine Barisano (Framingham)--> + +<p>Second, we contacted the <a href="https://alumni.northeastern.edu/about/alumni-center/">Northeastern University Alumni Center</a>. They were not open on Veterans&rsquo; Day 2019.</p> + +<p>Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: <a href="https://mtwyouth.org">More Than Words</a>.</p> + +<p>More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!</p> + +<blockquote> + <p>More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you&rsquo;ll see a mix of &ldquo;employees&rdquo; and &ldquo;volunteers&rdquo; helping out.</p></blockquote> + +<p>Booking was complicated, though, by the fact that Northeastern requires <a href="http://catalog.northeastern.edu/graduate/health-sciences/academic-policies-procedures/liability-insurance/">liability insurance</a> for all off-campus events. If <strong>you</strong> are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.</p> + +<p>Here are more facts about MTW:</p> + +<ul> + <li>the theater space seats 30 with lots of extra space</li> + <li>the two long tables in the reading room can seat about 20</li> + <li>the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor</li> + <li>there&rsquo;s no whiteboard, but MTW can supply an easel stand</li> + <li>much of the furniture in store is antique and for sale, so be careful using it &mdash; both to avoid damaging it, and because antiques can be oddly-shaped</li> + <li>the bookstore was closed on Veterans&rsquo; Day, but we were able to have our event and buy books</li> + <li>MTW may have fridge space to temporarily hold leftovers</li> + <li>closest T station: <a href="https://www.mbta.com/stops/place-tumnl">Tufts Medical Center</a></li></ul> + +<h3 id="keynote--none">Keynote = None</h3> + +<p>The original, ambitious plan was to invite two keynote speakers &mdash; one working in industry and one from academia &mdash; to enrich the offsite with new knowledge. And because this was the PRL&rsquo;s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)</p> + +<p>We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.</p> + +<p>Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).</p> +<!-- to future planners: we invited Kathi Fisler; she had grant-writing plans for that day and would be interested in coming to a future offsite--> + +<h3 id="food-coffee-tea">Food, Coffee, Tea</h3> + +<p><a href="https://flourbakery.com/">Flour Bakery + Cafe</a> provided lunch. For most days, you can order from Flour using an <a href="https://flourbakery.com/menu/catering/catering-information/">online form</a>. For holidays, you may need to send an email &mdash; that&rsquo;s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan &amp; gluten-free <em>everything spiced</em> salads.</p> + +<p><a href="https://www.trycuppacoffee.com/location/">Cuppacoffee</a> provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything &mdash; the order + napkins, milk, cream cheese, butter, and knives &mdash; ourselves because Cuppacoffee is very close to MTW.</p> + +<p><a href="http://www.cmartboston.com/">CMart</a> sold us water and juices. (There is also a Whole Foods near MTW.)</p> + +<p>At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour&rsquo;s <a href="https://youtu.be/kIbhckqanHI">sticky buns</a>.</p> + +<h3 id="event-program">Event Program</h3> + +<p>The following assumptions/goals constrained the schedule for the day:</p> + +<ul> + <li>give all 17 on-campus Ph.D. students a talk slot; talks must <strong>either</strong> communicate one technical idea from recent work <strong>or</strong> say what led the speaker to apply to a PL Ph.D. program</li> + <li>allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)</li> + <li>do not make the audience sit for more than 1 hour at a time</li> + <li>maximize the number and length of breaks (to encourage discussions)</li> + <li>include a relevant and fun welcome activity</li> + <li>save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions</li></ul> + +<p>We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.</p> + +<p>The activity was <em>codewalks</em>. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition/blob/master/src/main/java/com/seriouscompany/business/java/fizzbuzz/packagenamingpackage/impl/math/arithmetics/IntegerDivider.java">IntegerDivider</a> class from a silly FizzBuzz implementation and a solution to the <a href="https://en.wikipedia.org/wiki/Dutch_national_flag_problem">Dutch national flag problem</a>). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and <a href="https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/">egoless programming</a>.</p> + +<p>The talks ran conference-style. One organizer played &ldquo;session chair&rdquo; to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.</p> + +<p>For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.</p> + +<h2 id="looking-back-at-the-details">Looking Back at the Details</h2> + +<p>Reading Day in the Spring may be a good, free date for future retreats.</p> + +<p>We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.</p> + +<p>The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we&rsquo;d keep the morning breaks short &mdash; just enough to use the restroom and grab a coffee &mdash; and aim for 30-min breaks in the afternoon.</p> + +<p>Groups of three 15-minute talks worked well, but groups of four talks might be equally good.</p> + +<p>Perhaps each speaker should get the chance to pick a talk length. <a href="https://nepls.org/">NEPLS</a>, for example, allows a choice between 5-min. and 30-min. talks.</p> + +<p>The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick &ldquo;topics&rdquo; beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.</p> +<!-- to future organizers: the panel began asking faculty for a mistake in their career and wound up with a defensive flavor--> + +<h2 id="looking-back-overall">Looking Back, Overall</h2> + +<p>This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.</p> + +<p>There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time &mdash; keep the same date &amp; program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.</p> + +<p>With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.</p> \ No newline at end of file diff --git a/blog/feeds/offsite.rss.xml b/blog/feeds/offsite.rss.xml new file mode 100644 index 00000000..0790ce43 --- /dev/null +++ b/blog/feeds/offsite.rss.xml @@ -0,0 +1,546 @@ + + + + PRL Blog: Posts tagged 'offsite' + PRL Blog: Posts tagged 'offsite' + http://prl.ccs.neu.edu/blog/tags/offsite.html + Wed, 15 Jan 2020 12:16:35 UT + Wed, 15 Jan 2020 12:16:35 UT + 1800 + + The Typed Racket Optimizer vs. Transient + http://prl.ccs.neu.edu/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/?utm_source=offsite&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + Wed, 15 Jan 2020 12:16:35 UT + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> + + PRL Offsite 2019 Retrospective + http://prl.ccs.neu.edu/blog/2019/12/12/prl-offsite-2019-retrospective/?utm_source=offsite&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-12-12-prl-offsite-2019-retrospective + Thu, 12 Dec 2019 12:51:53 UT + Ben Greenman, Olek Gierczak + +<p>On November 11th 2019, the PRL had a private offsite meeting at the <a href="https://mtwyouth.org">More Than Words bookstore</a> in downtown Boston. For future offsite organizers, this post records what happened and how.</p> +<!-- more--> + +<h2 id="early-planning-goals">Early Planning, Goals</h2> + +<p>Every Fall, the PRL holds a kickoff meeting to assign <a href="http://prl.ccs.neu.edu/contact">roles</a> for the coming academic year. This year, Prof. Amal Ahmed led the meeting and expressed interest in having a retreat at some point. Seconds later, Amal enlisted Olek Gierczak and Ben Greenman to help plan a retreat.</p> + +<blockquote> + <p>Ben G. was on the (unsuccessful) 2018 retreat planning committee. The three lessons from that year were: (1) Veterans Day is a possible date in the Fall, (2) there are approximately zero possible dates in the Spring and Summer, (3) the <a href="http://www.warrencenter.com">Warren Conference Center</a> is <a href="https://www.northeastern.edu/events/northeastern-owned-off-campus-venues">Northeastern University&rsquo;s only franchised</a> off-campus venue.</p></blockquote> + +<p>The <strong>motivation</strong> for the retreat was to bring our growing department together for one day in the hope that everyone has (at least) a small interaction with everyone else. These little interactions are crucial for new Ph.D. students &mdash; both to get to know the group, and to become known. They&rsquo;re also useful for everyone else to build a stronger community and to open doors for collaboration. And yet, despite the fact that these interactions are an admittedly key benefit of having a lab, the last time the PRL got all together in a big way (more than a few hours for PL seminar or Ph.D. open house) was for the <a href="http://www.ccs.neu.edu/home/matthias/7480-s17/index.html">Spring 2017 edition</a> of HOPL.</p> + +<p>Our primary <strong>goal</strong> this year was for every Ph.D student to present research. Time permitting, we wanted a keynote speaker, a panel discussion, and activities. Weather permitting, we wanted to go outside during lunch.</p> + +<p>In light of the main goal, we prefer to call the event an &ldquo;offsite&rdquo; (or &ldquo;offsite meeting&rdquo;) rather than a &ldquo;retreat&rdquo; because the target was an informative day rather than a relaxing one.</p> + +<h2 id="booking-and-logistics">Booking and Logistics</h2> + +<p>Our planning went like this: (1) pick a date, (2) find a venue, (3) invite a keynote speaker, (4) order food, and (5) arrange the technical program. The process took 2 months because that&rsquo;s all we had.</p> + +<h3 id="date--nov-11">Date = Nov 11</h3> + +<p>In the beginning, we chose Veterans&rsquo; Day (2019&ndash;11&ndash;11) and Northeastern <a href="https://registrar.northeastern.edu/app/uploads/2019-2020-University-Wide-Calendar-List.pdf">reading day</a> (2019&ndash;12&ndash;05) as possible dates. We ended up with Veterans&rsquo; Day.</p> + +<p>A small number of lab members were opposed to Veterans&rsquo; Day. They gave two reasons: the Fall semester is especially busy, and Veterans&rsquo; Day is a federal holiday.</p> + +<h3 id="venue--mtw">Venue = MTW</h3> + +<p>The first venue we tried was the <a href="http://www.warrencenter.com">Warren Conference Center</a> in Framingham. The venue was difficult to contact. We submitted an online form; no response. We searched the website for contact email addresses; no luck. We called the office, got forwarded to the event manager, and left a message; no response. Lastly we asked the <a href="https://www.khoury.northeastern.edu/people/chelsea-smith/">Khoury Events Team</a> to reach out on our behalf. They returned with an email address and <a href="/blog/static/warren-center-meetings-and-retreats.pdf">event menu</a> (Thank you Chelsea and Donald!) and we decided the Warren Center was not a good fit for our small and short-notice offsite.</p> +<!-- Donald Pepple (Northeastern) made contact with Christine Barisano (Framingham)--> + +<p>Second, we contacted the <a href="https://alumni.northeastern.edu/about/alumni-center/">Northeastern University Alumni Center</a>. They were not open on Veterans&rsquo; Day 2019.</p> + +<p>Third, we turned to Google and Yelp for venue options in New England. Most were at a corporate-level price range, but this search led to our winner: <a href="https://mtwyouth.org">More Than Words</a>.</p> + +<p>More Than Words (MTW) is a bookstore and event space. We got in touch via email, visited the store, and then booked. Easy!</p> + +<blockquote> + <p>More Than Words is also a nonprofit social enterprise that offers job training for ~350 young adults each year. If you visit, you&rsquo;ll see a mix of &ldquo;employees&rdquo; and &ldquo;volunteers&rdquo; helping out.</p></blockquote> + +<p>Booking was complicated, though, by the fact that Northeastern requires <a href="http://catalog.northeastern.edu/graduate/health-sciences/academic-policies-procedures/liability-insurance/">liability insurance</a> for all off-campus events. If <strong>you</strong> are booking an event, get a contract from the venue well ahead of time and allow 2 to 4 weeks for Northeastern to process and sign it. We received our contract with 1 week until the payment was due and were very fortunate that the Northeastern administrative staff met the deadline.</p> + +<p>Here are more facts about MTW:</p> + +<ul> + <li>the theater space seats 30 with lots of extra space</li> + <li>the two long tables in the reading room can seat about 20</li> + <li>the projector is 16:9 native and accepts HDMI input; bring your own HDMI adaptor</li> + <li>there&rsquo;s no whiteboard, but MTW can supply an easel stand</li> + <li>much of the furniture in store is antique and for sale, so be careful using it &mdash; both to avoid damaging it, and because antiques can be oddly-shaped</li> + <li>the bookstore was closed on Veterans&rsquo; Day, but we were able to have our event and buy books</li> + <li>MTW may have fridge space to temporarily hold leftovers</li> + <li>closest T station: <a href="https://www.mbta.com/stops/place-tumnl">Tufts Medical Center</a></li></ul> + +<h3 id="keynote--none">Keynote = None</h3> + +<p>The original, ambitious plan was to invite two keynote speakers &mdash; one working in industry and one from academia &mdash; to enrich the offsite with new knowledge. And because this was the PRL&rsquo;s first offsite, it was decided that these speakers must have roughly-equal research overlap with every Ph.D. student. (Later meetings could specialize.)</p> + +<p>We failed to come up with many candidates that met our goal, and so we fell back to a simpler plan: pick one. To this end, we sent out a Google Form to assess preferences and research overlap. The form responses indicated a clear favorite, who we invited.</p> + +<p>Our chosen speaker, however, was unable to attend. Rather than invite a different guest, we replaced the keynote time with a morning activity (more on that later).</p> +<!-- to future planners: we invited Kathi Fisler; she had grant-writing plans for that day and would be interested in coming to a future offsite--> + +<h3 id="food-coffee-tea">Food, Coffee, Tea</h3> + +<p><a href="https://flourbakery.com/">Flour Bakery + Cafe</a> provided lunch. For most days, you can order from Flour using an <a href="https://flourbakery.com/menu/catering/catering-information/">online form</a>. For holidays, you may need to send an email &mdash; that&rsquo;s what we did, and the catering staff quickly helped us place an order. We ordered 16 assorted meat sandwiches, 16 assorted veggie sandwiches, 4 vegan hummus sandwiches, and 2 vegan &amp; gluten-free <em>everything spiced</em> salads.</p> + +<p><a href="https://www.trycuppacoffee.com/location/">Cuppacoffee</a> provided coffee, tea, and breakfast items. In the morning, that was: 3 gallons coffee, 1 gallon brewed black tea, 16 bagels, 12 muffins, and 10 croissants. In the afternoon, that was: 2 gallons coffee and 1 gallon brewed green tea. We were able to pick up everything &mdash; the order + napkins, milk, cream cheese, butter, and knives &mdash; ourselves because Cuppacoffee is very close to MTW.</p> + +<p><a href="http://www.cmartboston.com/">CMart</a> sold us water and juices. (There is also a Whole Foods near MTW.)</p> + +<p>At the end of the day, the leftovers included ~2 gallons of coffee and ~8 sweet potato sandwiches. People asked for more meat sandwiches, more blueberry muffins, and Flour&rsquo;s <a href="https://youtu.be/kIbhckqanHI">sticky buns</a>.</p> + +<h3 id="event-program">Event Program</h3> + +<p>The following assumptions/goals constrained the schedule for the day:</p> + +<ul> + <li>give all 17 on-campus Ph.D. students a talk slot; talks must <strong>either</strong> communicate one technical idea from recent work <strong>or</strong> say what led the speaker to apply to a PL Ph.D. program</li> + <li>allow at least 10 minutes per talk (because the audience may have trouble following a day of 5-minute talks, and the speakers may have trouble preparing informative 8-minute talks)</li> + <li>do not make the audience sit for more than 1 hour at a time</li> + <li>maximize the number and length of breaks (to encourage discussions)</li> + <li>include a relevant and fun welcome activity</li> + <li>save time for a panel discussion at the end, with everyone sitting around a room able to freely ask questions</li></ul> + +<p>We decided to start with an hour-long activity, split the day into hour-long blocks of three 15-minute talks (10 min. talk, 5 min. Q/A) and one 15-minute break, and end with a 1.5-hour panel discussion. In total, we started the activity at 9:25am and ended the panel at 6:40pm.</p> + +<p>The activity was <em>codewalks</em>. Before the offsite, we (the organizers) picked a few short and interesting programs (for example, the <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition/blob/master/src/main/java/com/seriouscompany/business/java/fizzbuzz/packagenamingpackage/impl/math/arithmetics/IntegerDivider.java">IntegerDivider</a> class from a silly FizzBuzz implementation and a solution to the <a href="https://en.wikipedia.org/wiki/Dutch_national_flag_problem">Dutch national flag problem</a>). During breakfast, we made 2-person teams and gave each team a printed copy of one program. Then, for the activity, we selected one team to present the code and three panelists from the audience to lead a discussion. Discussions ran for about 10 minutes, and then we picked another team; half the teams did not present. This activity ended up being fun (thanks to the great participants) and definitely met its serious goals; namely, to practice reading, speaking, critical thinking, and <a href="https://blog.codinghorror.com/the-ten-commandments-of-egoless-programming/">egoless programming</a>.</p> + +<p>The talks ran conference-style. One organizer played &ldquo;session chair&rdquo; to help each speaker set up and to announce each talk. Questions mid-talk were allowed, but most questions arrived after the speaker finished.</p> + +<p>For the panel discussion, we put chairs around the room and asked people to sit more-or-less comfortably. The panel moderator started by asking one question to the faculty, and we took things from there as answers arrived and inspired new questions.</p> + +<h2 id="looking-back-at-the-details">Looking Back at the Details</h2> + +<p>Reading Day in the Spring may be a good, free date for future retreats.</p> + +<p>We planned a 15-minute breakfast/welcome slot, but wound up needing 25 minutes to give people time to prepare for the activity.</p> + +<p>The planned 15-minute breaks often had to be cut short because talks ran longer. Next time, we&rsquo;d keep the morning breaks short &mdash; just enough to use the restroom and grab a coffee &mdash; and aim for 30-min breaks in the afternoon.</p> + +<p>Groups of three 15-minute talks worked well, but groups of four talks might be equally good.</p> + +<p>Perhaps each speaker should get the chance to pick a talk length. <a href="https://nepls.org/">NEPLS</a>, for example, allows a choice between 5-min. and 30-min. talks.</p> + +<p>The panel ended up too chaotic. People often had lots to say and it was difficult to queue questions during a speech. One idea for next time is to seat the professors together and give each a time limit to answer; this would organize the discussion, but may not improve the quality. Another idea is to pick &ldquo;topics&rdquo; beforehand and have the moderator enforce a time limit on each topic. Or maybe we should drop the panel unless we have a clear goal for what to discuss.</p> +<!-- to future organizers: the panel began asking faculty for a mistake in their career and wound up with a defensive flavor--> + +<h2 id="looking-back-overall">Looking Back, Overall</h2> + +<p>This was a good offsite. Organizing was mostly easy and straightforward. We very much enjoyed hearing what everyone had been working on.</p> + +<p>There were two rough parts to the organizing. First, we faced some difficult initial choices about the date, venue, and program. Second, we feel that we put undue pressure on students to prepare talks with short notice. Both challenges could easily be avoided next time &mdash; keep the same date &amp; program, and announce the plan early! Maybe though, a different program could lead to a more effective use of time.</p> + +<p>With all that in mind, we recommend having a similar meeting next year or next semester. It was extremely useful to sync up with the whole lab, good practice to make a 10-minute talk, and overall a rewarding (though stressful) day.</p> \ No newline at end of file diff --git a/blog/feeds/package.atom.xml b/blog/feeds/package.atom.xml new file mode 100644 index 00000000..fb176cee --- /dev/null +++ b/blog/feeds/package.atom.xml @@ -0,0 +1,362 @@ + + + PRL Blog: Posts tagged 'package' + + + urn:http-prl-ccs-neu-edu:-blog-tags-package-html + 2017-03-03T08:54:20Z + + PLT Redex: mf-apply + + urn:http-prl-ccs-neu-edu:-blog-2017-03-03-plt-redex-mf-apply + 2017-03-03T08:54:20Z + 2017-03-03T08:54:20Z + + Ben Greenman + +<p>The <code>mf-apply</code> keyword is for checked metafunction application in PLT Redex. In other words, <code>(mf-apply f x)</code> is just like <code>(f x)</code>, but errors if <code>f</code> is not a previously-defined metafunction.</p> + +<p>Also, consider applying to attend <em>The Racket School of Semantics and Languages</em> in Salt Lake City this summer: <a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></p> +<!-- more--> + +<h2 id="metafunctions-vs-list-patterns">Metafunctions vs. List Patterns</h2> + +<p>Have you used PLT Redex? Good! Maybe this has happened to you:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span> +<span class="normal">25</span> +<span class="normal">26</span> +<span class="normal">27</span> +<span class="normal">28</span> +<span class="normal">29</span> +<span class="normal">30</span> +<span class="normal">31</span> +<span class="normal">32</span> +<span class="normal">33</span> +<span class="normal">34</span> +<span class="normal">35</span> +<span class="normal">36</span> +<span class="normal">37</span> +<span class="normal">38</span> +<span class="normal">39</span> +<span class="normal">40</span> +<span class="normal">41</span> +<span class="normal">42</span> +<span class="normal">43</span> +<span class="normal">44</span> +<span class="normal">45</span> +<span class="normal">46</span> +<span class="normal">47</span> +<span class="normal">48</span> +<span class="normal">49</span> +<span class="normal">50</span> +<span class="normal">51</span> +<span class="normal">52</span> +<span class="normal">53</span> +<span class="normal">54</span> +<span class="normal">55</span> +<span class="normal">56</span> +<span class="normal">57</span> +<span class="normal">58</span> +<span class="normal">59</span> +<span class="normal">60</span> +<span class="normal">61</span> +<span class="normal">62</span> +<span class="normal">63</span> +<span class="normal">64</span> +<span class="normal">65</span> +<span class="normal">66</span> +<span class="normal">67</span> +<span class="normal">68</span> +<span class="normal">69</span> +<span class="normal">70</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 1. You define a language</span> +<span class="p">(</span><span class="n">define-language</span> <span class="n">STLC</span> + <span class="p">[</span><span class="n">V</span> <span class="n">::=</span> <span class="n">integer</span> <span class="n">boolean</span> <span class="n">C</span><span class="p">]</span> + <span class="p">[</span><span class="n">C</span> <span class="n">::=</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">Λ</span> <span class="n">::=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)]</span> + <span class="p">[</span><span class="n">M</span> <span class="n">::=</span> <span class="p">(</span><span class="n">M</span> <span class="n">M</span><span class="p">)</span> <span class="n">V</span> <span class="n">Λ</span> <span class="n">x</span><span class="p">]</span> + <span class="p">[</span><span class="n">τ</span> <span class="n">::=</span> <span class="n">Int</span> <span class="n">Bool</span> <span class="p">(</span><span class="n">τ</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">ρ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">V</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">Γ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">τ</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">x</span> <span class="n">::=</span> <span class="n">variable-not-otherwise-mentioned</span><span class="p">]</span> + <span class="kd">#:binding-forms</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span> <span class="kd">#:refers-to</span> <span class="n">x</span><span class="p">))</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 2. You define a few metafunctions</span> +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;lam</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">Λ</span> + <span class="p">[(</span><span class="n">closure-&gt;lam</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">Λ</span><span class="p">])</span> + +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;env</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">ρ</span> + <span class="p">[(</span><span class="n">closure-&gt;env</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">ρ</span><span class="p">])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 3. You try defining a judgment form . . .</span> +<span class="p">(</span><span class="n">define-judgment-form</span> <span class="n">STLC</span> + <span class="kd">#:mode</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">I</span> <span class="n">O</span><span class="p">)</span> + <span class="kd">#:contract</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Var</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">x</span> <span class="p">(</span><span class="n">x</span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_0</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_1</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">---</span> <span class="n">FVS-App</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">M_0</span> <span class="n">M_1</span><span class="p">)</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x_0</span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)</span> <span class="n">Λ</span><span class="p">)</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-remove))" style="color: inherit">set-remove</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="n">x_0</span><span class="p">)))</span> + <span class="n">---</span> <span class="n">FVS-Λ</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Integer</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">integer_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Boolean</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">boolean_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">closure-&gt;lam</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-subtract))" style="color: inherit">set-subtract</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))))</span> + <span class="n">---</span> <span class="n">FVS-Closure</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">C</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 4. You test the judgment, and it mysteriously fails</span> +<span class="p">(</span><span class="n">judgment-holds</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">closure</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">())</span> + <span class="p">()))</span> +<span class="c1">;; ==&gt; #f</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><strong>WHAT HAPPENED??!</strong></p> + +<p>The problem is this line in the <code>FVS-Closure</code> rule:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which checks that the list <code>(closure-env C)</code> (whose first element is the symbol <code>closure-env</code> and second element is the symbol <code>C</code>) matches the pattern <code>((x_1 τ_1) ...)</code>.</p> + +<p>Right.</p> + +<p>Of course you meant to apply the metafunction <code>closure-&gt;env</code> but made a typo. And since the syntax for metafunction application is the same as the syntax for building a list, Redex doesn&rsquo;t report an error.</p> + +<p>We can fix this code with the new <a href="https://www.cs.utah.edu/plt/snapshots/current/doc/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._mf-apply%29%29"><code>mf-apply</code></a> keyword (available on <a href="https://github.com/racket/racket">GitHub</a> or in a <a href="https://www.cs.utah.edu/plt/snapshots/">snapshot build</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running <code>raco make</code> now gives a compile-time error.</p> + +<pre><code> term: expected a previously defined metafunction + at: closure-env + in: (mf-apply closure-env C)</code></pre> + +<h3 id="but-i-still-need-to-type-mf-apply-correctly">But I still need to type <code>mf-apply</code> correctly!</h3> + +<p>Leif Andersen says:</p> + +<blockquote> + <p>I should point out that this has the issue of you still need to type <code>mf-apply</code> correctly. ;)</p></blockquote> + +<p>That is, if you accidentally write:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-applu</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then the code compiles, thinking you intend to match a list of three elements against the pattern.</p> + +<p>Never fear, there are at least two solutions.</p> + +<h4 id="solution-1-rename-mf-apply">Solution 1: rename <code>mf-apply</code></h4> + +<p>A simple fix is to rename the <code>mf-apply</code> keyword to something shorter (and harder to mis-type):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._rename-in))" style="color: inherit">rename-in</a></span> <span class="n">redex</span> + <span class="p">[</span><span class="n">mf-apply</span> <span class="n">MF</span><span class="p">]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="solution-2-the-mf-apply-lang-extension">Solution 2: the <code>mf-apply</code> lang extension</h4> + +<p>A fancier solution is to install the <code>mf-apply</code> meta-language.</p> + +<pre><code> $ raco pkg install mf-apply</code></pre> + +<p>This language updates the <a href="http://docs.racket-lang.org/reference/readtables.html#%28tech._readtable%29"><em>readtable</em></a> to interpret S-expressions that begin with <code>#{</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="o">#</span><span class="p">{</span><span class="ss">f</span> <span class="ss">x</span> <span class="ss"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>as a metafunction application:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">f</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>You the programmer only needs to write the <code>#{....}</code> syntax.</p> + +<p>Source code is on GitHub:</p> + +<ul> + <li><a href="https://github.com/bennn/mf-apply">https://github.com/bennn/mf-apply</a></li></ul> + +<p>(It&rsquo;s the simplest lang-extension I know of)</p> + +<h2 id="what-is-plt-redex">What is PLT Redex?</h2> + +<p>PLT Redex is a library for semantics engineering. One of my favorite Redex features is it implements capture-avoiding substitution and α-equivalence for any language with a <code>#:binding-forms</code> specification (such as STLC, above).</p> + +<p>You can read more:</p> + +<ul> + <li>in the &ldquo;Amb&rdquo; tutorial: <a href="http://docs.racket-lang.org/redex/tutorial.html">http://docs.racket-lang.org/redex/tutorial.html</a></li> + <li>in the &ldquo;Long Tutorial&rdquo;: <a href="http://docs.racket-lang.org/redex/redex2015.html">http://docs.racket-lang.org/redex/redex2015.html</a></li> + <li>in the Redex reference manual: <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html">http://docs.racket-lang.org/redex/The_Redex_Reference.html</a></li> + <li>on the PLT Redex website: <a href="https://redex.racket-lang.org/">https://redex.racket-lang.org/</a></li> + <li>on GitHub: <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + +<p>And if you act now, you can become a <em>Redexan</em> between July 10 and July 14 at the summer school in Salt Lake City, Utah:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/package.rss.xml b/blog/feeds/package.rss.xml new file mode 100644 index 00000000..df3618b3 --- /dev/null +++ b/blog/feeds/package.rss.xml @@ -0,0 +1,362 @@ + + + + PRL Blog: Posts tagged 'package' + PRL Blog: Posts tagged 'package' + http://prl.ccs.neu.edu/blog/tags/package.html + Fri, 03 Mar 2017 08:54:20 UT + Fri, 03 Mar 2017 08:54:20 UT + 1800 + + PLT Redex: mf-apply + http://prl.ccs.neu.edu/blog/2017/03/03/plt-redex-mf-apply/?utm_source=package&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-03-03-plt-redex-mf-apply + Fri, 03 Mar 2017 08:54:20 UT + Ben Greenman + +<p>The <code>mf-apply</code> keyword is for checked metafunction application in PLT Redex. In other words, <code>(mf-apply f x)</code> is just like <code>(f x)</code>, but errors if <code>f</code> is not a previously-defined metafunction.</p> + +<p>Also, consider applying to attend <em>The Racket School of Semantics and Languages</em> in Salt Lake City this summer: <a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></p> +<!-- more--> + +<h2 id="metafunctions-vs-list-patterns">Metafunctions vs. List Patterns</h2> + +<p>Have you used PLT Redex? Good! Maybe this has happened to you:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span> +<span class="normal">24</span> +<span class="normal">25</span> +<span class="normal">26</span> +<span class="normal">27</span> +<span class="normal">28</span> +<span class="normal">29</span> +<span class="normal">30</span> +<span class="normal">31</span> +<span class="normal">32</span> +<span class="normal">33</span> +<span class="normal">34</span> +<span class="normal">35</span> +<span class="normal">36</span> +<span class="normal">37</span> +<span class="normal">38</span> +<span class="normal">39</span> +<span class="normal">40</span> +<span class="normal">41</span> +<span class="normal">42</span> +<span class="normal">43</span> +<span class="normal">44</span> +<span class="normal">45</span> +<span class="normal">46</span> +<span class="normal">47</span> +<span class="normal">48</span> +<span class="normal">49</span> +<span class="normal">50</span> +<span class="normal">51</span> +<span class="normal">52</span> +<span class="normal">53</span> +<span class="normal">54</span> +<span class="normal">55</span> +<span class="normal">56</span> +<span class="normal">57</span> +<span class="normal">58</span> +<span class="normal">59</span> +<span class="normal">60</span> +<span class="normal">61</span> +<span class="normal">62</span> +<span class="normal">63</span> +<span class="normal">64</span> +<span class="normal">65</span> +<span class="normal">66</span> +<span class="normal">67</span> +<span class="normal">68</span> +<span class="normal">69</span> +<span class="normal">70</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 1. You define a language</span> +<span class="p">(</span><span class="n">define-language</span> <span class="n">STLC</span> + <span class="p">[</span><span class="n">V</span> <span class="n">::=</span> <span class="n">integer</span> <span class="n">boolean</span> <span class="n">C</span><span class="p">]</span> + <span class="p">[</span><span class="n">C</span> <span class="n">::=</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">Λ</span> <span class="n">::=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)]</span> + <span class="p">[</span><span class="n">M</span> <span class="n">::=</span> <span class="p">(</span><span class="n">M</span> <span class="n">M</span><span class="p">)</span> <span class="n">V</span> <span class="n">Λ</span> <span class="n">x</span><span class="p">]</span> + <span class="p">[</span><span class="n">τ</span> <span class="n">::=</span> <span class="n">Int</span> <span class="n">Bool</span> <span class="p">(</span><span class="n">τ</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ</span><span class="p">)]</span> + <span class="p">[</span><span class="n">ρ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">V</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">Γ</span> <span class="n">::=</span> <span class="p">((</span><span class="n">x</span> <span class="n">τ</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)]</span> + <span class="p">[</span><span class="n">x</span> <span class="n">::=</span> <span class="n">variable-not-otherwise-mentioned</span><span class="p">]</span> + <span class="kd">#:binding-forms</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span> <span class="kd">#:refers-to</span> <span class="n">x</span><span class="p">))</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 2. You define a few metafunctions</span> +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;lam</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">Λ</span> + <span class="p">[(</span><span class="n">closure-&gt;lam</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">Λ</span><span class="p">])</span> + +<span class="p">(</span><span class="n">define-metafunction</span> <span class="n">STLC</span> + <span class="n">closure-&gt;env</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">C</span> <span class="k"><a href="http://docs.racket-lang.org/reference/function-contracts.html#(form._((lib._racket/contract/base..rkt)._-~3e))" style="color: inherit">-&gt;</a></span> <span class="n">ρ</span> + <span class="p">[(</span><span class="n">closure-&gt;env</span> <span class="p">(</span><span class="n">closure</span> <span class="n">Λ</span> <span class="n">ρ</span><span class="p">))</span> + <span class="n">ρ</span><span class="p">])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 3. You try defining a judgment form . . .</span> +<span class="p">(</span><span class="n">define-judgment-form</span> <span class="n">STLC</span> + <span class="kd">#:mode</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">I</span> <span class="n">O</span><span class="p">)</span> + <span class="kd">#:contract</span> <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Var</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">x</span> <span class="p">(</span><span class="n">x</span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_0</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M_1</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">---</span> <span class="n">FVS-App</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">M_0</span> <span class="n">M_1</span><span class="p">)</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x_0</span> <span class="n">τ</span><span class="p">)</span> <span class="n">M</span><span class="p">)</span> <span class="n">Λ</span><span class="p">)</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">M</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-remove))" style="color: inherit">set-remove</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="n">x_0</span><span class="p">)))</span> + <span class="n">---</span> <span class="n">FVS-Λ</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Integer</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">integer_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="n">---</span> <span class="n">FVS-Boolean</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">boolean_0</span> <span class="p">())]</span> + <span class="p">[</span> + <span class="p">(</span><span class="n">where</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">closure-&gt;lam</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">Λ</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="p">(</span><span class="n">where</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="o">,</span><span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sets.html#(def._((lib._racket/set..rkt)._set-subtract))" style="color: inherit">set-subtract</a></span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_0</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> <span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">x_1</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))))</span> + <span class="n">---</span> <span class="n">FVS-Closure</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="n">C</span> <span class="p">(</span><span class="n">x_2</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))])</span> + + +<span class="c1">;; -----------------------------------------------------------------------------</span> +<span class="c1">;; 4. You test the judgment, and it mysteriously fails</span> +<span class="p">(</span><span class="n">judgment-holds</span> + <span class="p">(</span><span class="n">free-variables</span> <span class="p">(</span><span class="n">closure</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">())</span> + <span class="p">()))</span> +<span class="c1">;; ==&gt; #f</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><strong>WHAT HAPPENED??!</strong></p> + +<p>The problem is this line in the <code>FVS-Closure</code> rule:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which checks that the list <code>(closure-env C)</code> (whose first element is the symbol <code>closure-env</code> and second element is the symbol <code>C</code>) matches the pattern <code>((x_1 τ_1) ...)</code>.</p> + +<p>Right.</p> + +<p>Of course you meant to apply the metafunction <code>closure-&gt;env</code> but made a typo. And since the syntax for metafunction application is the same as the syntax for building a list, Redex doesn&rsquo;t report an error.</p> + +<p>We can fix this code with the new <a href="https://www.cs.utah.edu/plt/snapshots/current/doc/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._mf-apply%29%29"><code>mf-apply</code></a> keyword (available on <a href="https://github.com/racket/racket">GitHub</a> or in a <a href="https://www.cs.utah.edu/plt/snapshots/">snapshot build</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running <code>raco make</code> now gives a compile-time error.</p> + +<pre><code> term: expected a previously defined metafunction + at: closure-env + in: (mf-apply closure-env C)</code></pre> + +<h3 id="but-i-still-need-to-type-mf-apply-correctly">But I still need to type <code>mf-apply</code> correctly!</h3> + +<p>Leif Andersen says:</p> + +<blockquote> + <p>I should point out that this has the issue of you still need to type <code>mf-apply</code> correctly. ;)</p></blockquote> + +<p>That is, if you accidentally write:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="n">....</span> + <span class="p">(</span><span class="n">where</span> <span class="p">((</span><span class="n">x_1</span> <span class="n">τ_1</span><span class="p">)</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="p">(</span><span class="n">mf-applu</span> <span class="n">closure-env</span> <span class="n">C</span><span class="p">))</span> + <span class="n">....</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Then the code compiles, thinking you intend to match a list of three elements against the pattern.</p> + +<p>Never fear, there are at least two solutions.</p> + +<h4 id="solution-1-rename-mf-apply">Solution 1: rename <code>mf-apply</code></h4> + +<p>A simple fix is to rename the <code>mf-apply</code> keyword to something shorter (and harder to mis-type):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._rename-in))" style="color: inherit">rename-in</a></span> <span class="n">redex</span> + <span class="p">[</span><span class="n">mf-apply</span> <span class="n">MF</span><span class="p">]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="solution-2-the-mf-apply-lang-extension">Solution 2: the <code>mf-apply</code> lang extension</h4> + +<p>A fancier solution is to install the <code>mf-apply</code> meta-language.</p> + +<pre><code> $ raco pkg install mf-apply</code></pre> + +<p>This language updates the <a href="http://docs.racket-lang.org/reference/readtables.html#%28tech._readtable%29"><em>readtable</em></a> to interpret S-expressions that begin with <code>#{</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="o">#</span><span class="p">{</span><span class="ss">f</span> <span class="ss">x</span> <span class="ss"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>as a metafunction application:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">mf-apply</span> <span class="n">racket</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="n">redex</span><span class="p">)</span> + +<span class="p">(</span><span class="n">term</span> <span class="p">(</span><span class="n">mf-apply</span> <span class="n">f</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>You the programmer only needs to write the <code>#{....}</code> syntax.</p> + +<p>Source code is on GitHub:</p> + +<ul> + <li><a href="https://github.com/bennn/mf-apply">https://github.com/bennn/mf-apply</a></li></ul> + +<p>(It&rsquo;s the simplest lang-extension I know of)</p> + +<h2 id="what-is-plt-redex">What is PLT Redex?</h2> + +<p>PLT Redex is a library for semantics engineering. One of my favorite Redex features is it implements capture-avoiding substitution and α-equivalence for any language with a <code>#:binding-forms</code> specification (such as STLC, above).</p> + +<p>You can read more:</p> + +<ul> + <li>in the &ldquo;Amb&rdquo; tutorial: <a href="http://docs.racket-lang.org/redex/tutorial.html">http://docs.racket-lang.org/redex/tutorial.html</a></li> + <li>in the &ldquo;Long Tutorial&rdquo;: <a href="http://docs.racket-lang.org/redex/redex2015.html">http://docs.racket-lang.org/redex/redex2015.html</a></li> + <li>in the Redex reference manual: <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html">http://docs.racket-lang.org/redex/The_Redex_Reference.html</a></li> + <li>on the PLT Redex website: <a href="https://redex.racket-lang.org/">https://redex.racket-lang.org/</a></li> + <li>on GitHub: <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + +<p>And if you act now, you can become a <em>Redexan</em> between July 10 and July 14 at the summer school in Salt Lake City, Utah:</p> + +<ul> + <li><a href="http://summer-school.racket-lang.org/2017/">http://summer-school.racket-lang.org/2017/</a></li></ul> \ No newline at end of file diff --git a/blog/feeds/performance.atom.xml b/blog/feeds/performance.atom.xml new file mode 100644 index 00000000..4f445e70 --- /dev/null +++ b/blog/feeds/performance.atom.xml @@ -0,0 +1,188 @@ + + + PRL Blog: Posts tagged 'performance' + + + urn:http-prl-ccs-neu-edu:-blog-tags-performance-html + 2018-05-08T15:37:37Z + + Sampling Gradual Typing Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + 2018-05-08T15:37:37Z + 2018-05-08T15:37:37Z + Ben Greenman + Zeina Migeed + + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> + + A few cores too many + + urn:http-prl-ccs-neu-edu:-blog-2016-08-03-a-few-cores-too-many + 2016-08-03T14:09:02Z + 2016-08-03T14:09:02Z + + Ben Greenman + +<p>Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.</p> +<!-- more--> + +<p>Last year, we proposed a method for evaluating the performance of gradual type systems based on measuring <em>every possible configuration</em> of typed and untyped code that a programmer might explore <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">(pdf)</a>. Given the freedom that gradual typing offers, this is the only realistic way to measure the performance of a gradual type system.</p> + +<p>But it is a lot to measure! While developing the method, we spent over 3 months benchmarking a total of 75,844 configurations. Each configuration is a complete program and some gradual typings caused some programs to slow by 50x or even 100x, so many of these configurations took minutes to run.</p> + +<p>The next question we asked was naturally &ldquo;how can we scale this method to large software projects?&rdquo; In <a href="http://docs.racket-lang.org/ts-reference/Libraries_Provided_With_Typed_Racket.html#%28part._.Porting_.Untyped_.Modules_to_.Typed_.Racket%29">our case</a>, the number of gradually typed configurations scaled exponentially with the number of modules. Current gradual type system for <a href="https://github.com/mvitousek/reticulated">Python</a> and <a href="http://www.di.ens.fr/~zappa/readings/ecoop15.pdf">JavaScript</a> are exponential in the number of <em>variables</em> in the program.</p> + +<p>We explored two solutions:</p> + +<ol> + <li>Max New began work on a prediction model (inspired by work on <a href="http://subs.emis.de/LNI/Proceedings/Proceedings213/185.pdf">software product lines</a>) to estimate the performance of <code>2^N</code> configurations after polynomially-many measurements.</li> + <li>Asumu Takikawa and I shopped for a multi-core computer.</li></ol> + +<p>By Thanksgiving, we had bought a Linux machine with 2 <a href="http://www.amd.com/en-us/products/server/opteron/6000/6300">AMD Opteron 6376 2.3GHz</a> processors (16 cores each) and put it to work running benchmarks on 29 cores simultaneously. Life was good.</p> + +<p>Later that winter, Max implemented a prediction algorithm. The basic idea was to focus on <em>boundaries</em> between modules and isolate their effect on performance. If two modules are untyped, their boundary will have zero cost. If the same two modules are typed, their boundary might result in an overall performance improvement due to type-driven optimizations. And if one module is typed and the other untyped, their boundary will suffer some cost of type checking at runtime. In general a program with <code>N</code> modules has at most <code>N(N - 1) / 2</code> internal boundaries, so it is far more time-efficient to measure only the boundaries than to benchmark <code>2^N</code> gradually typed configurations.</p> + +<p>Fast-forward to March, we had a prototype prediction algorithm and it was time to test. Again using 29 cores (because, why not), we gathered cost/benefit numbers for one 4-module benchmark and used them to predict performance for its 16 configurations. The results were not very good.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-1.png" alt="Figure 1: True running time vs. predicted running time for 16 configurations" /> + <p class="caption">Figure 1: True running time vs. predicted running time for 16 configurations</p></div> + +<p>Those green circles are the ground truth, the average running time after 5 iterations of each config. The blue triangles are what we predicted. Except for configurations 0 and 8, the triangles are FAR off from the truth. Many are even negative &hellip; obviously the algorithm needs work.</p> + +<p>But then, out of frustration, desperation, or just good luck, Max compared the predictions to ground truth data gathered on a <em>single</em> core, leaving the other 31 cores idle.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-2.png" alt="Figure 2: Predictions made using measurements from a single core" /> + <p class="caption">Figure 2: Predictions made using measurements from a single core</p></div> + +<p>First off, the red &ldquo;sequential truth&rdquo; dots are slightly closer to the predicted triangles. Second &mdash; and this is the scary part &mdash; the red dots are very different from the green dots. <em>Running on 1 core vs. 29 cores should not change the measurements!</em></p> + +<p>From here we tried increasing the running time of the benchmark, removing I/O and system calls, checking for hyperthreading (ARM cores don&rsquo;t support it), and even changing the cores&rsquo; CPU governor. The hope was that results taken from 1 core could match results from <code>N</code> cores, for some <code>N &gt; 1</code>. It turns out <code>N = 2</code> was stable, but even for <code>N = 3</code> we found graphs like the following:</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-3.png" alt="Figure 3: exact running times. Same-colored dots in each column should be tightly clustered." /> + <p class="caption">Figure 3: exact running times. Same-colored dots in each column should be tightly clustered.</p></div> + +<p>This data is for the same 16 configurations as the previous two graphs. Green dots are exact running times measured with 25 cores. Red dots are running times measured with 1 core. The red dots are much closer together, and always unimodal. The green dots are evidence that maybe the 32-core machine has, as Jan Vitek put it, 30 cores too many.</p> + +<blockquote> + <p>&ldquo;Oh my. You think it&rsquo;ll never happen to you. Well, now I&rsquo;ve learned my lesson.&rdquo;</p><!-- bg: If anyone knows this quote I will be AMAZED. If anyone can even Google this quote, I'll buy them 2 beers and a pizza.--></blockquote> + +<p>And so, we said goodbye to the last 4 months of data and started over running at most two cores. The new results are all stable, but still we keep pinching ourselves.</p> + +<p>P.S. the results from <a href="http://www.ccs.neu.edu/racket/pubs/#popl16-tfgnvf">POPL 2016</a> are just fine, as they were not taken on the new machine running more than 2 cores. If you have time to confirm, that data is in our <a href="http://www.ccs.neu.edu/home/asumu/artifacts/popl-2016/">artifact</a> and in the <a href="https://github.com/nuprl/gradual-typing-performance/tree/master/paper/popl-2016/data">gradual-typing-performance</a> repo.</p> \ No newline at end of file diff --git a/blog/feeds/performance.rss.xml b/blog/feeds/performance.rss.xml new file mode 100644 index 00000000..1b66ffca --- /dev/null +++ b/blog/feeds/performance.rss.xml @@ -0,0 +1,184 @@ + + + + PRL Blog: Posts tagged 'performance' + PRL Blog: Posts tagged 'performance' + http://prl.ccs.neu.edu/blog/tags/performance.html + Tue, 08 May 2018 15:37:37 UT + Tue, 08 May 2018 15:37:37 UT + 1800 + + Sampling Gradual Typing Performance + http://prl.ccs.neu.edu/blog/2018/05/08/sampling-gradual-typing-performance/?utm_source=performance&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + Tue, 08 May 2018 15:37:37 UT + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> + + A few cores too many + http://prl.ccs.neu.edu/blog/2016/08/03/a-few-cores-too-many/?utm_source=performance&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-08-03-a-few-cores-too-many + Wed, 03 Aug 2016 14:09:02 UT + Ben Greenman + +<p>Performance matters for software systems, but performance is not always easy to measure. At the PRL we recently had a scare with some unreliable measurements. Here is the story.</p> +<!-- more--> + +<p>Last year, we proposed a method for evaluating the performance of gradual type systems based on measuring <em>every possible configuration</em> of typed and untyped code that a programmer might explore <a href="http://www.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf">(pdf)</a>. Given the freedom that gradual typing offers, this is the only realistic way to measure the performance of a gradual type system.</p> + +<p>But it is a lot to measure! While developing the method, we spent over 3 months benchmarking a total of 75,844 configurations. Each configuration is a complete program and some gradual typings caused some programs to slow by 50x or even 100x, so many of these configurations took minutes to run.</p> + +<p>The next question we asked was naturally &ldquo;how can we scale this method to large software projects?&rdquo; In <a href="http://docs.racket-lang.org/ts-reference/Libraries_Provided_With_Typed_Racket.html#%28part._.Porting_.Untyped_.Modules_to_.Typed_.Racket%29">our case</a>, the number of gradually typed configurations scaled exponentially with the number of modules. Current gradual type system for <a href="https://github.com/mvitousek/reticulated">Python</a> and <a href="http://www.di.ens.fr/~zappa/readings/ecoop15.pdf">JavaScript</a> are exponential in the number of <em>variables</em> in the program.</p> + +<p>We explored two solutions:</p> + +<ol> + <li>Max New began work on a prediction model (inspired by work on <a href="http://subs.emis.de/LNI/Proceedings/Proceedings213/185.pdf">software product lines</a>) to estimate the performance of <code>2^N</code> configurations after polynomially-many measurements.</li> + <li>Asumu Takikawa and I shopped for a multi-core computer.</li></ol> + +<p>By Thanksgiving, we had bought a Linux machine with 2 <a href="http://www.amd.com/en-us/products/server/opteron/6000/6300">AMD Opteron 6376 2.3GHz</a> processors (16 cores each) and put it to work running benchmarks on 29 cores simultaneously. Life was good.</p> + +<p>Later that winter, Max implemented a prediction algorithm. The basic idea was to focus on <em>boundaries</em> between modules and isolate their effect on performance. If two modules are untyped, their boundary will have zero cost. If the same two modules are typed, their boundary might result in an overall performance improvement due to type-driven optimizations. And if one module is typed and the other untyped, their boundary will suffer some cost of type checking at runtime. In general a program with <code>N</code> modules has at most <code>N(N - 1) / 2</code> internal boundaries, so it is far more time-efficient to measure only the boundaries than to benchmark <code>2^N</code> gradually typed configurations.</p> + +<p>Fast-forward to March, we had a prototype prediction algorithm and it was time to test. Again using 29 cores (because, why not), we gathered cost/benefit numbers for one 4-module benchmark and used them to predict performance for its 16 configurations. The results were not very good.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-1.png" alt="Figure 1: True running time vs. predicted running time for 16 configurations" /> + <p class="caption">Figure 1: True running time vs. predicted running time for 16 configurations</p></div> + +<p>Those green circles are the ground truth, the average running time after 5 iterations of each config. The blue triangles are what we predicted. Except for configurations 0 and 8, the triangles are FAR off from the truth. Many are even negative &hellip; obviously the algorithm needs work.</p> + +<p>But then, out of frustration, desperation, or just good luck, Max compared the predictions to ground truth data gathered on a <em>single</em> core, leaving the other 31 cores idle.</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-2.png" alt="Figure 2: Predictions made using measurements from a single core" /> + <p class="caption">Figure 2: Predictions made using measurements from a single core</p></div> + +<p>First off, the red &ldquo;sequential truth&rdquo; dots are slightly closer to the predicted triangles. Second &mdash; and this is the scary part &mdash; the red dots are very different from the green dots. <em>Running on 1 core vs. 29 cores should not change the measurements!</em></p> + +<p>From here we tried increasing the running time of the benchmark, removing I/O and system calls, checking for hyperthreading (ARM cores don&rsquo;t support it), and even changing the cores&rsquo; CPU governor. The hope was that results taken from 1 core could match results from <code>N</code> cores, for some <code>N &gt; 1</code>. It turns out <code>N = 2</code> was stable, but even for <code>N = 3</code> we found graphs like the following:</p> + +<div class="figure"><img src="/img/a-few-cores-too-many-3.png" alt="Figure 3: exact running times. Same-colored dots in each column should be tightly clustered." /> + <p class="caption">Figure 3: exact running times. Same-colored dots in each column should be tightly clustered.</p></div> + +<p>This data is for the same 16 configurations as the previous two graphs. Green dots are exact running times measured with 25 cores. Red dots are running times measured with 1 core. The red dots are much closer together, and always unimodal. The green dots are evidence that maybe the 32-core machine has, as Jan Vitek put it, 30 cores too many.</p> + +<blockquote> + <p>&ldquo;Oh my. You think it&rsquo;ll never happen to you. Well, now I&rsquo;ve learned my lesson.&rdquo;</p><!-- bg: If anyone knows this quote I will be AMAZED. If anyone can even Google this quote, I'll buy them 2 beers and a pizza.--></blockquote> + +<p>And so, we said goodbye to the last 4 months of data and started over running at most two cores. The new results are all stable, but still we keep pinching ourselves.</p> + +<p>P.S. the results from <a href="http://www.ccs.neu.edu/racket/pubs/#popl16-tfgnvf">POPL 2016</a> are just fine, as they were not taken on the new machine running more than 2 cores. If you have time to confirm, that data is in our <a href="http://www.ccs.neu.edu/home/asumu/artifacts/popl-2016/">artifact</a> and in the <a href="https://github.com/nuprl/gradual-typing-performance/tree/master/paper/popl-2016/data">gradual-typing-performance</a> repo.</p> \ No newline at end of file diff --git a/blog/feeds/pliss.atom.xml b/blog/feeds/pliss.atom.xml new file mode 100644 index 00000000..725b65f8 --- /dev/null +++ b/blog/feeds/pliss.atom.xml @@ -0,0 +1,70 @@ + + + PRL Blog: Posts tagged 'pliss' + + + urn:http-prl-ccs-neu-edu:-blog-tags-pliss-html + 2017-06-05T15:47:59Z + + Report: PLISS 2017 + + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-report-pliss-2017 + 2017-06-05T15:47:59Z + 2017-06-05T15:47:59Z + + Ming-Ho Yee + +<p>Two weeks ago, I attended the first <a href="https://pliss2017.github.io/">Programming Language Implementation Summer School</a>, held in beautiful Bertinoro, Italy.</p> + +<p>The goal of PLISS was &ldquo;to prepare early graduate students and advanced undergraduates for research in the field,&rdquo; and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.</p> +<!-- more--> + +<h2 id="the-talks">The Talks</h2> + +<p>With the amount of technical content at PLISS, there was easily something for everyone. <a href="http://janvitek.org/">Jan Vitek</a> and <a href="http://tratt.net/laurie/">Laurence Tratt</a> gave lectures that included hands-on exercises where we worked on JITs. <a href="https://www.cs.purdue.edu/homes/suresh/">Suresh Jagannathan</a> dived into the operational semantics of a distributed system, so we could reason about different weak consistency models. Francesco Logozzo gave us a whirlwind tour of abstract interpretation.</p> + +<p>Most of my favorite talks included some form of extra content, such as exercises, live-coding presentations, or demos. I found it really helpful to write actual code and apply what I had just learned, or to look at some concrete examples. The examples and exercises also helped with the pacing, as actively listening to four 90-minute talks every day is exhausting!</p> + +<p>Off the top of my head, these were some of my favorite talks:</p> + +<ul> + <li> + <p><strong>Dynamic Programming Language Implementation with LLVM</strong>, by Petr Maj, Oli Flückiger, and <a href="http://janvitek.org/">Jan Vitek</a>. As the first talk of the summer school, this was a gentle introduction for the rest of the week. We had <a href="https://github.com/PRL-PRG/pliss-rift/">exercises</a> (with intentional bugs to make us think!), and also brief overviews of intermediate languages, static analysis, and garbage collection. These three topics would later show up in more detail.</p></li> + <li> + <p><strong>Micro Virtual Machines</strong>, by <a href="http://users.cecs.anu.edu.au/~steveb/">Steve Blackburn</a>. This talk covered background information on virtual machines, and also the <a href="http://microvm.github.io/">Micro VM</a> project that Steve&rsquo;s group has been working on. A lot of the material was already familiar to me, but I still enjoyed the talk, and even got a few ideas for the project I&rsquo;m working on!</p></li> + <li> + <p><strong>Static Analysis</strong>, by <a href="http://matt.might.net/">Matt Might</a>. Matt&rsquo;s talk was based on one of his <a href="http://matt.might.net/articles/intro-static-analysis/">articles</a> and an older talk he&rsquo;s given. Impressively, the entire example was live-coded, with only a single mistake!</p></li> + <li> + <p><strong>Testing Language Implementations</strong>, by <a href="http://multicore.doc.ic.ac.uk/">Alastair Donaldson</a>. This was an entertaining talk, since Ally showed multiple examples of crashing compilers, and causing other kinds of mischief by triggering compiler bugs.</p></li></ul> + +<p>If you&rsquo;re disappointed that you couldn&rsquo;t see these talks, don&rsquo;t worry! The talks were recorded and will be posted very shortly.</p> + +<h2 id="the-people">The People</h2> + +<p>But there&rsquo;s more to PLISS than the talks. I&rsquo;m referring to <em>networking</em>, or the opportunity to get out and talk to other people about research.</p> + +<p>As an early graduate student, I&rsquo;ve been given a lot of advice about talking to people at conferences and the importance of the &ldquo;hallway track.&rdquo; I still have difficulty doing this at an actual conference, like <a href="http://pldi17.sigplan.org/home">PLDI</a> or <a href="http://2017.ecoop.org/">ECOOP</a>. When there are hundreds of attendees, or when people already know each other and are in conversation groups, I find it difficult to approach them.</p> + +<p>This was not the case at PLISS. There were fewer attendees: about fifty students and a dozen speakers. There was a good mix of undergraduate, master&rsquo;s, first-year PhD, and more senior PhD students. All our breakfasts, lunches, and breaks were together, so we would see the same people again and again, and inevitably start to learn each other&rsquo;s names. The speakers would also be among us, and there was a good ratio of speakers to students for discussions and mealtime mentoring.</p> + +<p>I had many opportunities to practice my &ldquo;research pitch.&rdquo; I talked to senior students and got advice. I talked to junior students and gave advice. Two different people I talked to about my research pointed me to the same paper to read. I found another student who was working with <a href="http://research.cs.wisc.edu/wpis/papers/popl95.pdf">IFDS</a>, an algorithm I have spent much time trying to understand. And, one day at lunch, my table discovered that we were all working on static analysis!</p> + +<p>As much as I enjoyed the talks, I think the best part of PLISS was meeting and talking to other people. You can replace talks with videos (but you lose the speaker-audience interaction), and you can replace conversations with other forms of communication. But there isn&rsquo;t really anything that can replace the serendipity of bumping into someone with a shared interest.</p> + +<h2 id="the-location">The Location</h2> + +<p>Actually, the <em>other</em> best part of PLISS was the location. Italy is a beautiful country with delicious food. And Bertinoro is a small town on the top of a hill, with a breathtaking view of the surrounding area. The lectures were held in a <a href="https://pliss2017.github.io/images/pics/7.jpg">castle at the top of the hill</a> (photo credit: Steve Blackburn). The speakers lived in the castle for the week, while the students lived in the former monastery (seems fitting), which has been renovated into a university residence.</p> + +<p>Here are my two favorite pictures I took (click for full size):</p> + +<p><a href="/img/pliss2017-1.jpg"><img src="/img/pliss2017-1-thumb.jpg" alt="View from the castle" /></a></p> + +<p><a href="/img/pliss2017-2.jpg"><img src="/img/pliss2017-2-thumb.jpg" alt="Panorama" /></a></p> + +<p>Steve Blackburn has more pictures posted on the <a href="https://pliss2017.github.io/">PLISS website</a>.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>PLISS was a wonderful event. Many thanks need to be given to the speakers, organizers, and sponsors, for making this possible!</p> + +<p>If and when there is a second PLISS, I highly encourage students to apply! You will learn a lot from the lectures, from talking to the speakers, and meeting other students. And if it&rsquo;s in Bertinoro again, you can enjoy the weather and nice view!</p> \ No newline at end of file diff --git a/blog/feeds/pliss.rss.xml b/blog/feeds/pliss.rss.xml new file mode 100644 index 00000000..324a816d --- /dev/null +++ b/blog/feeds/pliss.rss.xml @@ -0,0 +1,70 @@ + + + + PRL Blog: Posts tagged 'pliss' + PRL Blog: Posts tagged 'pliss' + http://prl.ccs.neu.edu/blog/tags/pliss.html + Mon, 05 Jun 2017 15:47:59 UT + Mon, 05 Jun 2017 15:47:59 UT + 1800 + + Report: PLISS 2017 + http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/?utm_source=pliss&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-06-05-report-pliss-2017 + Mon, 05 Jun 2017 15:47:59 UT + Ming-Ho Yee + +<p>Two weeks ago, I attended the first <a href="https://pliss2017.github.io/">Programming Language Implementation Summer School</a>, held in beautiful Bertinoro, Italy.</p> + +<p>The goal of PLISS was &ldquo;to prepare early graduate students and advanced undergraduates for research in the field,&rdquo; and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.</p> +<!-- more--> + +<h2 id="the-talks">The Talks</h2> + +<p>With the amount of technical content at PLISS, there was easily something for everyone. <a href="http://janvitek.org/">Jan Vitek</a> and <a href="http://tratt.net/laurie/">Laurence Tratt</a> gave lectures that included hands-on exercises where we worked on JITs. <a href="https://www.cs.purdue.edu/homes/suresh/">Suresh Jagannathan</a> dived into the operational semantics of a distributed system, so we could reason about different weak consistency models. Francesco Logozzo gave us a whirlwind tour of abstract interpretation.</p> + +<p>Most of my favorite talks included some form of extra content, such as exercises, live-coding presentations, or demos. I found it really helpful to write actual code and apply what I had just learned, or to look at some concrete examples. The examples and exercises also helped with the pacing, as actively listening to four 90-minute talks every day is exhausting!</p> + +<p>Off the top of my head, these were some of my favorite talks:</p> + +<ul> + <li> + <p><strong>Dynamic Programming Language Implementation with LLVM</strong>, by Petr Maj, Oli Flückiger, and <a href="http://janvitek.org/">Jan Vitek</a>. As the first talk of the summer school, this was a gentle introduction for the rest of the week. We had <a href="https://github.com/PRL-PRG/pliss-rift/">exercises</a> (with intentional bugs to make us think!), and also brief overviews of intermediate languages, static analysis, and garbage collection. These three topics would later show up in more detail.</p></li> + <li> + <p><strong>Micro Virtual Machines</strong>, by <a href="http://users.cecs.anu.edu.au/~steveb/">Steve Blackburn</a>. This talk covered background information on virtual machines, and also the <a href="http://microvm.github.io/">Micro VM</a> project that Steve&rsquo;s group has been working on. A lot of the material was already familiar to me, but I still enjoyed the talk, and even got a few ideas for the project I&rsquo;m working on!</p></li> + <li> + <p><strong>Static Analysis</strong>, by <a href="http://matt.might.net/">Matt Might</a>. Matt&rsquo;s talk was based on one of his <a href="http://matt.might.net/articles/intro-static-analysis/">articles</a> and an older talk he&rsquo;s given. Impressively, the entire example was live-coded, with only a single mistake!</p></li> + <li> + <p><strong>Testing Language Implementations</strong>, by <a href="http://multicore.doc.ic.ac.uk/">Alastair Donaldson</a>. This was an entertaining talk, since Ally showed multiple examples of crashing compilers, and causing other kinds of mischief by triggering compiler bugs.</p></li></ul> + +<p>If you&rsquo;re disappointed that you couldn&rsquo;t see these talks, don&rsquo;t worry! The talks were recorded and will be posted very shortly.</p> + +<h2 id="the-people">The People</h2> + +<p>But there&rsquo;s more to PLISS than the talks. I&rsquo;m referring to <em>networking</em>, or the opportunity to get out and talk to other people about research.</p> + +<p>As an early graduate student, I&rsquo;ve been given a lot of advice about talking to people at conferences and the importance of the &ldquo;hallway track.&rdquo; I still have difficulty doing this at an actual conference, like <a href="http://pldi17.sigplan.org/home">PLDI</a> or <a href="http://2017.ecoop.org/">ECOOP</a>. When there are hundreds of attendees, or when people already know each other and are in conversation groups, I find it difficult to approach them.</p> + +<p>This was not the case at PLISS. There were fewer attendees: about fifty students and a dozen speakers. There was a good mix of undergraduate, master&rsquo;s, first-year PhD, and more senior PhD students. All our breakfasts, lunches, and breaks were together, so we would see the same people again and again, and inevitably start to learn each other&rsquo;s names. The speakers would also be among us, and there was a good ratio of speakers to students for discussions and mealtime mentoring.</p> + +<p>I had many opportunities to practice my &ldquo;research pitch.&rdquo; I talked to senior students and got advice. I talked to junior students and gave advice. Two different people I talked to about my research pointed me to the same paper to read. I found another student who was working with <a href="http://research.cs.wisc.edu/wpis/papers/popl95.pdf">IFDS</a>, an algorithm I have spent much time trying to understand. And, one day at lunch, my table discovered that we were all working on static analysis!</p> + +<p>As much as I enjoyed the talks, I think the best part of PLISS was meeting and talking to other people. You can replace talks with videos (but you lose the speaker-audience interaction), and you can replace conversations with other forms of communication. But there isn&rsquo;t really anything that can replace the serendipity of bumping into someone with a shared interest.</p> + +<h2 id="the-location">The Location</h2> + +<p>Actually, the <em>other</em> best part of PLISS was the location. Italy is a beautiful country with delicious food. And Bertinoro is a small town on the top of a hill, with a breathtaking view of the surrounding area. The lectures were held in a <a href="https://pliss2017.github.io/images/pics/7.jpg">castle at the top of the hill</a> (photo credit: Steve Blackburn). The speakers lived in the castle for the week, while the students lived in the former monastery (seems fitting), which has been renovated into a university residence.</p> + +<p>Here are my two favorite pictures I took (click for full size):</p> + +<p><a href="/img/pliss2017-1.jpg"><img src="/img/pliss2017-1-thumb.jpg" alt="View from the castle" /></a></p> + +<p><a href="/img/pliss2017-2.jpg"><img src="/img/pliss2017-2-thumb.jpg" alt="Panorama" /></a></p> + +<p>Steve Blackburn has more pictures posted on the <a href="https://pliss2017.github.io/">PLISS website</a>.</p> + +<h2 id="final-thoughts">Final Thoughts</h2> + +<p>PLISS was a wonderful event. Many thanks need to be given to the speakers, organizers, and sponsors, for making this possible!</p> + +<p>If and when there is a second PLISS, I highly encourage students to apply! You will learn a lot from the lectures, from talking to the speakers, and meeting other students. And if it&rsquo;s in Bertinoro again, you can enjoy the weather and nice view!</p> \ No newline at end of file diff --git a/blog/feeds/proceedings.atom.xml b/blog/feeds/proceedings.atom.xml new file mode 100644 index 00000000..0b9350d5 --- /dev/null +++ b/blog/feeds/proceedings.atom.xml @@ -0,0 +1,30 @@ + + + PRL Blog: Posts tagged 'proceedings' + + + urn:http-prl-ccs-neu-edu:-blog-tags-proceedings-html + 2016-06-13T10:50:14Z + + Does anyone still care about printed proceedings? (Grab some at NEU this week!) + + urn:http-prl-ccs-neu-edu:-blog-2016-06-13-does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week + 2016-06-13T10:50:14Z + 2016-06-13T10:50:14Z + + Gabriel Scherer + +<p>Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!</p> +<!-- more--> + +<p>If you are in the area and are interested, feel free to send me an email and come grab them. We have ASPLOS from XIII to XX, PLDI from 2005 to 2015 (but not 2014), and OOPSLA from 2002 to 2015. When I saw the stack, I grabbed the POPL ones from 2002 to 2015, but in fact I have no idea what to do with them and I&rsquo;m a bit skeptical they would be of use to me; if you know you would use them, I would be glad to let you have them.</p> + +<p>If you were to buy those proceedings at conference-subscription rates today, it would cost you a small fortune. Yet nobody seems to want them. An odd disconnect, that I found amusing and maybe worthy of a blog post.</p> + +<p>But don&rsquo;t get me wrong, the future of printed proceedings is not an important question. We should rather be asking ourselves: why are the products of the work of our communities not easily accessible in an Open Access long-term archive? Are you submitting your articles to arXiv, or another archive? Why not?</p> + +<p>Not caring about printed proceedings is perfectly fine; but please care about people outside institutions that want to access your work &mdash; for example, master student myself.</p> + +<hr /> + +<p><em>Update (August 2017):</em> Gabriel returned to France and left the POPL proceedings at his desk. The proceedings are sitting in the hallway at NEU, in case anyone cares.</p> \ No newline at end of file diff --git a/blog/feeds/proceedings.rss.xml b/blog/feeds/proceedings.rss.xml new file mode 100644 index 00000000..dc645f3c --- /dev/null +++ b/blog/feeds/proceedings.rss.xml @@ -0,0 +1,30 @@ + + + + PRL Blog: Posts tagged 'proceedings' + PRL Blog: Posts tagged 'proceedings' + http://prl.ccs.neu.edu/blog/tags/proceedings.html + Mon, 13 Jun 2016 10:50:14 UT + Mon, 13 Jun 2016 10:50:14 UT + 1800 + + Does anyone still care about printed proceedings? (Grab some at NEU this week!) + http://prl.ccs.neu.edu/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/?utm_source=proceedings&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-13-does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week + Mon, 13 Jun 2016 10:50:14 UT + Gabriel Scherer + +<p>Are you interested in printed conference Proceedings? We have a good stack of them left away at Northeastern University (Boston, MA) and it seems that nobody wants them!</p> +<!-- more--> + +<p>If you are in the area and are interested, feel free to send me an email and come grab them. We have ASPLOS from XIII to XX, PLDI from 2005 to 2015 (but not 2014), and OOPSLA from 2002 to 2015. When I saw the stack, I grabbed the POPL ones from 2002 to 2015, but in fact I have no idea what to do with them and I&rsquo;m a bit skeptical they would be of use to me; if you know you would use them, I would be glad to let you have them.</p> + +<p>If you were to buy those proceedings at conference-subscription rates today, it would cost you a small fortune. Yet nobody seems to want them. An odd disconnect, that I found amusing and maybe worthy of a blog post.</p> + +<p>But don&rsquo;t get me wrong, the future of printed proceedings is not an important question. We should rather be asking ourselves: why are the products of the work of our communities not easily accessible in an Open Access long-term archive? Are you submitting your articles to arXiv, or another archive? Why not?</p> + +<p>Not caring about printed proceedings is perfectly fine; but please care about people outside institutions that want to access your work &mdash; for example, master student myself.</p> + +<hr /> + +<p><em>Update (August 2017):</em> Gabriel returned to France and left the POPL proceedings at his desk. The proceedings are sitting in the hallway at NEU, in case anyone cares.</p> \ No newline at end of file diff --git a/blog/feeds/r.atom.xml b/blog/feeds/r.atom.xml new file mode 100644 index 00000000..b04d4e96 --- /dev/null +++ b/blog/feeds/r.atom.xml @@ -0,0 +1,849 @@ + + + PRL Blog: Posts tagged 'r' + + + urn:http-prl-ccs-neu-edu:-blog-tags-r-html + 2019-09-10T11:00:00Z + + Four Kinds of Scoping in R + + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-four-kinds-of-scoping-in-r + 2019-09-10T11:00:00Z + 2019-09-10T11:00:00Z + + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">first</a> and <a href="/blog/2019/09/10/scoping-in-r/">second</a> parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.</p> + +<p>In this third and final part of my blog series, I&rsquo;d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.</p> + +<p>This is a &ldquo;bonus&rdquo; blog post, because I&rsquo;m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.</p> +<!-- more--> + +<h2 id="lexical-scope-and-statistical-computation">Lexical scope and statistical computation</h2> + +<p>In <em>Lexical Scope and Statistical Computation</em>,<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">1</a></sup> Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.</p> + +<p>For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:</p> + +<ul> + <li><em>trivial</em>: no free variables allowed</li> + <li><em>static</em>: a free variable takes its value from a set of global variables</li> + <li><em>lexical</em>: a free variable takes the value of the binding that was in effect when the function was defined</li> + <li><em>dynamic</em>: a free variable takes the value of the most recent assignment to that variable</li></ul> + +<p>Note that under this set of definitions, <em>static scoping</em> is a separate scoping rule and not another name for <em>lexical scoping</em>.</p> + +<p>It is possible to simulate each of strategies in R. For fun, we can even construct &ldquo;factories&rdquo; that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)</p> + +<h3 id="template">Template</h3> + +<p>Our examples will follow the template given below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">factory</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="o">&lt;???&gt;</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">factory</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># error, 0, 1, or 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We want to define a <code>factory</code> that takes a function literal and returns a closure that implements the desired scoping rule.</p> + +<p>Our example consists of three definitions of <code>x</code>. On line 5, we assign <code>0</code> to <code>x</code> at the top level. On line 7, we assign <code>1</code> to <code>x</code> inside function <code>h</code>, where we also create the closure. On line 12, we assign <code>2</code> to <code>x</code> inside the function <code>f</code> and right before we call <code>g</code>, which is the closure.</p> + +<p>Finally, we call <code>f</code> and observe the result:</p> + +<ul> + <li>Under trivial scoping, no free variables are allowed, so <code>f()</code> should result in an error.</li> + <li>Under static scoping, free variables may only refer to global variables, so <code>f()</code> should return <code>0</code>.</li> + <li>Under lexical scoping, free variables refer to the variables in scope when the function was defined, so <code>f()</code> should return <code>1</code>.</li> + <li>Under dynamic scoping, free variables take the value from the most recent assignment, so <code>f()</code> should return <code>2</code>.</li></ul> + +<p>We will implement the body of <code>factory</code> in only 3&ndash;5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to <code>factory</code> on line 10.</p> + +<h3 id="trivial-scoping">Trivial scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeTrivial</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">baseenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeTrivial</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># Error in f(0) : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>substitute</code> returns the unevaluated parse tree for <code>fun</code>. In other words, it obtains the literal argument that was passed for <code>fun</code>. This works because of call-by-need semantics in R: function arguments are packaged up into <em>promises</em>. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">2</a></sup> discusses laziness in R in more detail.</p> + +<p>In this example, on line 8, we call <code>factory</code> with <code>function(a) x+a</code> as the argument for the formal parameter <code>fun</code>. Then, we evaluate that parse tree with <code>eval</code>.</p> + +<p>At this point, <code>res</code> is the closure with expression <code>function(a) x+a</code> and a reference to the environment of <code>makeTrivial</code>. On line 3, we change that reference to <code>baseenv()</code>, which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.</p> + +<p>Therefore, variable lookup in the function literal will only search the base environment, so <code>f()</code> results in an error.</p> + +<h3 id="static-scoping">Static scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeStatic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">globalenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeStatic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, on line 3, we update the environment of <code>res</code> to refer to <code>globalenv()</code>, which is the top-level environment where globals are defined.</p> + +<p>Therefore, variable lookup searches the top-level environment, so <code>f()</code> returns <code>0</code>.</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeLexical</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeLexical</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of <code>res</code> to <code>parent.frame()</code>, which is the environment of the function (<code>h</code>) that called the current function (<code>makeLexical</code>). This allows us to simulate lexical scoping, as if the function literal was evaluated inside <code>h</code>, rather than <code>makeLexical</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>h</code>, so <code>f()</code> returns <code>1</code>.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeDynamic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">function</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="nf">res</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> + <span class="p">}</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeDynamic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, we need another level of indirection. <code>makeDynamic</code> returns an anonymous function literal. The anonymous function takes <code>...</code>, which represents an arbitrary list of arguments, and then on line 5 we call <code>res</code> with those exact arguments. Note that we set the environment of <code>res</code> to be the environment of the <em>caller</em> of the anonymous function. Because of the multiple levels of indirection, the caller is <code>f</code>, on line 17.</p> + +<p>On line 12, <code>makeDynamic</code> returns a closure for the anonymous function. <code>h</code> returns that closure when it is called, and assigns it to <code>g</code>. When <code>g</code> is called on line 17, the function literal <code>function(a) x+a</code> is finally evaluated, and its environment is set to the environment of <code>f</code>, the caller of <code>g</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>f</code>, so <code>f()</code> returns <code>2</code>.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the <a href="/blog/2019/09/10/scoping-in-r/">previous post</a>, it&rsquo;s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.</p> + +<p>And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.</p> + +<p><em>I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>A. Goel and J. Vitek. &ldquo;On the Design, Implementation and Use of Laziness in R,&rdquo; in <em>Proceedings of the ACM in Programming Languages (PACMPL)</em>, vol. 3, no. OOPSLA, 2019. To appear. [<a href="http://janvitek.org/pubs/oopsla19a.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">↩</a></p></li></ol></div> + + Scoping in R + + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-scoping-in-r + 2019-09-10T10:00:00Z + 2019-09-10T10:00:00Z + + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">previous post</a> of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: <em>is R lexically or dynamically scoped?</em></p> +<!-- more--> + +<p>Recall the example program from before:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s examine what happens when we run this example. First, we create a mapping for <code>x</code> in the top-level environment. On line 2, we define a function <code>f</code>, which returns the value of some <code>x</code>. On line 3, we define a function <code>g</code>, which creates a new mapping for <code>x</code>, and then calls <code>f</code>. Note that the assignment on line 4 does <em>not</em> update the definition on line 1.</p> + +<p>When <code>f</code> is called, it needs to look up the value of <code>x</code>. In other words, does the reference of <code>x</code> on line 2 refer to the assignment on line 1 or the assignment on line 4? If <code>f</code> returns <code>1</code>, then the behaviour matches lexical scoping. If it returns <code>2</code>, then the behaviour matches dynamic scoping.</p> + +<p>When we run this example, the result is <code>1</code>. This implies that R is lexically scoped.</p> + +<p>But there&rsquo;s more to this story. In the rest of this blog post, I&rsquo;ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.</p> + +<p>The <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">next and final part</a> of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.</p> + +<h2 id="r-is-lexically-scoped-but">R is lexically scoped, but&hellip;</h2> + +<p>In <em>Evaluating the Design of the R Language</em>,<sup><a href="#2019-09-10-scoping-in-r-footnote-1-definition" name="2019-09-10-scoping-in-r-footnote-1-return">1</a></sup> Morandat, Hill, Osvald, and Vitek write:</p> + +<blockquote> + <p>As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.</p></blockquote> + +<p>In other words, R provides many different &ldquo;escape hatches&rdquo;&mdash;ways to bypass lexical scoping. Additionally, even without escape hatches, some of R&rsquo;s functionality can be surprising.</p> + +<h3 id="functions-environments-and-variables-in-r">Functions, environments, and variables in R</h3> + +<p>Before we look at some examples, I think it&rsquo;s useful to briefly discuss some of the core concepts in R that relate to scoping.</p> + +<ul> + <li> + <p><strong>Functions.</strong> R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.</p></li> + <li> + <p><strong>Environments.</strong> An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the &ldquo;enclosing&rdquo; environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.</p></li> + <li> + <p><strong>Variable lookup.</strong> When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.</p></li> + <li> + <p><strong>Variable assignment.</strong> <code>&lt;-</code> is the variable assignment operator in R. The expression <code>x &lt;- 1</code> assigns the value <code>1</code> to the variable <code>x</code> in the current environment. If a mapping for <code>x</code> already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.</p></li></ul> + +<p>From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.<sup><a href="#2019-09-10-scoping-in-r-footnote-2-definition" name="2019-09-10-scoping-in-r-footnote-2-return">2</a></sup></p> + +<p>On the other hand, variable lookup depends on the run-time state of the program&mdash;names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.</p> + +<p>The following examples will make this clear.</p> + +<h3 id="examples">Examples</h3> + +<h4 id="adding-variable-mappings-at-run-time">Adding variable mappings at run time</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it creates a function <code>g</code> that returns <code>x</code>, assigns <code>2</code> to <code>x</code>, and then calls <code>g</code>. When <code>g</code> is called, it looks up <code>x</code>. Since no mapping is found in <code>g</code>&rsquo;s environment, it searches in the enclosing environment (<code>f</code>&rsquo;s), and finds that <code>x</code> has value <code>2</code>. Therefore, <code>g</code> returns <code>2</code>.</p> + +<p>Note that the <code>x</code> on line 3 is resolved only when function <code>g</code> is called, not when it is defined. However, when <code>g</code> is defined, its environment has a reference to <code>f</code>&rsquo;s environment. Therefore, as long as <code>x</code> is defined <em>before</em> <code>g</code> is called, the lookup will always succeed.</p> + +<p>Here&rsquo;s a second example:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>f</code> is a function that branches on its argument, <code>b</code>. If <code>b</code> evaluates to true, then the expression <code>x &lt;- 2</code> is evaluated, and a mapping for <code>x</code> is created in <code>f</code>&rsquo;s environment. Otherwise, no mapping is created.</p> + +<p>When we look up the value of <code>x</code> on line 5, R will first search the function&rsquo;s environment. If <code>b</code> evaluated to true, then R will find a value for <code>x</code>, which is <code>2</code>. Otherwise, R will search in the enclosing environment of <code>f</code>, and find that <code>x</code> is <code>1</code>.</p> + +<p>Both of these examples vaguely resemble dynamic scoping, in that <code>x</code> takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.</p> + +<h4 id="function-lookup">Function lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> <span class="c1"># not an error</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">(</span><span class="m">42</span><span class="p">)</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.</p> + +<p>In this example, we call <code>g</code> with the argument <code>42</code>, which is not a function. Then, in the body of <code>g</code>, we call <code>f(0)</code> on line 3, which requires looking up <code>f</code>. Although there is an <code>f</code> in the environment of <code>g</code>, its value is <code>42</code>, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so <code>f(0)</code> returns <code>0</code>.</p> + +<p>This behaviour exists because <code>c</code> is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.</p> + +<h4 id="super-assignment">Super assignment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="n">x</span> <span class="o">&lt;&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +<span class="n">x</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>&lt;&lt;-</code> is the &ldquo;super assignment&rdquo; operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.</p> + +<p>In the above program, we define <code>x</code> to be <code>0</code> at the top level, and then define the function <code>f</code>. When we call <code>f</code> on line 7, it assigns <code>1</code> to <code>x</code> on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, <code>f</code> returns <code>x</code>, which is looked up from the local environment and has value <code>1</code>. Finally, line 8 looks up <code>x</code> from the top level environment, which has value <code>2</code>.</p> + +<h4 id="evaluating-arbitrary-code">Evaluating arbitrary code</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">eval</span><span class="p">(</span><span class="nf">parse</span><span class="p">(</span><span class="n">text</span> <span class="o">=</span> <span class="n">t</span><span class="p">))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="s">"x &lt;- 0"</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument <code>t</code>, which happens to be the string <code>"x &lt;- 0"</code>. Then, when line 4 executes, the lookup of <code>x</code> returns <code>0</code>.</p> + +<h4 id="simulating-dynamic-scope">Simulating dynamic scope</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.frame</span><span class="p">())</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 3, we perform an explicit variable lookup for <code>x</code>, but we do so in the environment <code>parent.frame()</code>, which refers to the calling function&rsquo;s environment, in this case, <code>g</code>&rsquo;s environment.. Therefore, the lookup returns <code>2</code>.</p> + +<p>Note that R has a similarly named function, <code>parent.env(e)</code> which returns the <em>enclosing environment</em> of the given environment <code>e</code>.</p> + +<h4 id="constructing-an-arbitrary-environment">Constructing an arbitrary environment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">e</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> + <span class="n">e</span><span class="o">$</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">3</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="n">e</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 3</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it constructs a new environment, <code>e</code>, which is initially empty. (By default, its enclosing environment is the current environment, which is <code>f</code>&rsquo;s.) Next, on line 4, it directly adds a mapping to that environment, assigning <code>3</code> to <code>x</code>. Then, on line 5, the lookup is explicitly done in environment <code>e</code>, so <code>f</code> returns <code>3</code>.</p> + +<h4 id="deleting-mappings">Deleting mappings</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">rm</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.env</span><span class="p">(</span><span class="nf">environment</span><span class="p">()))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># Error in f() : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Not only is it possible to dynamically add and modify mappings in R, but it is also possible to <em>delete</em> mappings. This is what line 3 does: it explicitly removes the mapping for <code>x</code> from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when <code>f</code> is called, the lookup of <code>x</code> fails and an error is raised.</p> + +<h4 id="infinite-loop-during-variable-lookup">Infinite loop during variable lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">enva</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="n">envb</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">enva</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">envb</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">envb</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="nf">environment</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="nf">f</span><span class="p">()</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.</p> + +<p>On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: <code>enva</code>&rsquo;s enclosing environment is <code>envb</code>, and <code>envb</code>&rsquo;s enclosing environment is <code>enva</code>.</p> + +<p>On line 5, we define a function with a free variable, <code>x</code>, but on line 6, we set <code>f</code>&rsquo;s environment to be <code>enva</code>. Finally, we call <code>f</code>.</p> + +<p>When the body of <code>f</code> is evaluated, it needs to look up <code>x</code>. Lookup starts in <code>f</code>&rsquo;s environment, which we set to be <code>enva</code>. Since no mapping for <code>x</code> is found, lookup continues in <code>enva</code>&rsquo;s enclosing environment, which is <code>envb</code>. However, <code>envb</code> is also empty, so lookup continues in its enclosing environment, which is <code>enva</code>, and now lookup results in an infinite loop.</p> + +<h3 id="an-intuition-for-scoping-in-r">An intuition for scoping in R</h3> + +<p>Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1"># example 1</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> + +<span class="c1"># example 2</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>It seems that <code>x</code> takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What&rsquo;s missing from our intuition?</p> + +<p>The key insight is that R is <em>function scoped</em>. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <code>&lt;-</code> <em>does not create a new scope</em>, and it is more useful to think of it as a mutation <em>on the current environment</em>. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)</p> + +<p>In a sense, it might be more accurate to say that R <em>environments</em> are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.</p> + +<h2 id="discussion">Discussion</h2> + +<p>All of this might make you a little uncomfortable, and uncertain about R&rsquo;s scoping rules.</p> + +<p>On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply &ldquo;lexical-like&rdquo; behaviour, and the creators have confirmed that lexical scoping was the intent.</p> + +<p>On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment&mdash;but this is not consistent with R&rsquo;s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>This ambiguity shows up in a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-3-definition" name="2019-09-10-scoping-in-r-footnote-3-return">3</a></sup> where the authors write:</p> + +<blockquote> + <p>Furthermore, because variable scoping in R is dynamic and can be modified at the language level [&hellip;] it cannot be trivially guaranteed that <code>x</code> is going to point to the same data structure throughout the entire execution of the loop.</p></blockquote> + +<p>It is true that a variable <code>x</code> may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.</p> + +<p>It is true that variable <em>lookup</em> is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of <em>dynamic scope</em>, then it would be fair to say that R is dynamically scoped.</p> + +<p>But if your definition of <em>dynamic scope</em> is &ldquo;a variable is bound to the most recent assignment during the program&rsquo;s execution,&rdquo; then it is not correct to say R is dynamically scoped.</p> + +<p>I think we have this ambiguity because <em>scope</em> (the places in a program where a variable can be referenced) and <em>variable lookup</em> or <em>name resolution</em> (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.</p> + +<p>Personally, I prefer the definition of <em>scope</em> that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is <em>both</em> lexically and dynamically scoped, or that R is <em>neither</em> lexically and dynamically scoped.</p> + +<p>I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language&mdash;when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.</p> + +<p>Regardless, it is very easy to miscommunicate, so I think it&rsquo;s important to be very clear and make sure you and your audience know what definitions of scoping you&rsquo;re using!</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>This entire adventure started when we were working on a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-4-definition" name="2019-09-10-scoping-in-r-footnote-4-return">4</a></sup> and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!</p> + +<p>This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.</p> + +<p>To summarize, I would say that <em>scope</em> refers to the places in a program where a variable is visible and can be referenced. Under <em>lexical scoping</em>, the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program. Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment during the program&rsquo;s execution.</p> + +<p>I would say that R <em>aims</em> to be lexically scoped&mdash;it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!</p> + +<p>If you want more content about R and scoping, the <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">third and final part</a> of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.</p> + +<p><strong>Edited 2020/02/21:</strong> For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), <a href="http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/">this blog post</a> has some nice examples and diagrams.</p> + +<p><em>I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.</em></p> + +<p><em>If you liked this post, you may also be interested in the following Twitter threads about R: <a href="https://twitter.com/mhyee/status/1063983175163158531">one</a>, <a href="https://twitter.com/mhyee/status/1067818720532316166">two</a> and <a href="https://twitter.com/mhyee/status/1074744049951739905">three</a>.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>F. Morandat, B. Hill, L. Osvald, J. Vitek. &ldquo;Evaluating the Design of the R Language,&rdquo; in <em>Proceedings of the European Conference on Object-Oriented Programming (ECOOP)</em>, 2012. [<a href="https://doi.org/10.1007/978-3-642-31057-7_6">DOI</a>][<a href="http://janvitek.org/pubs/ecoop12.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. &ldquo;Lexical Scope and Statistical Computing&rdquo;, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-2-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-3-definition" class="footnote-definition"> + <p>L. Stadler, A. Welc, C. Humer, and M. Jordan. &ldquo;Optimizing R Language Execution via Aggressive Speculation,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2016. [<a href="https://doi.org/10.1145/2989225.2989236">DOI</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-3-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-4-definition" class="footnote-definition"> + <p>O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. &ldquo;R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2019. To appear. [<a href="http://janvitek.org/pubs/dls19.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-4-return">↩</a></p></li></ol></div> \ No newline at end of file diff --git a/blog/feeds/r.rss.xml b/blog/feeds/r.rss.xml new file mode 100644 index 00000000..ab1829b9 --- /dev/null +++ b/blog/feeds/r.rss.xml @@ -0,0 +1,847 @@ + + + + PRL Blog: Posts tagged 'r' + PRL Blog: Posts tagged 'r' + http://prl.ccs.neu.edu/blog/tags/r.html + Tue, 10 Sep 2019 11:00:00 UT + Tue, 10 Sep 2019 11:00:00 UT + 1800 + + Four Kinds of Scoping in R + http://prl.ccs.neu.edu/blog/2019/09/10/four-kinds-of-scoping-in-r/?utm_source=r&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-four-kinds-of-scoping-in-r + Tue, 10 Sep 2019 11:00:00 UT + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">first</a> and <a href="/blog/2019/09/10/scoping-in-r/">second</a> parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.</p> + +<p>In this third and final part of my blog series, I&rsquo;d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.</p> + +<p>This is a &ldquo;bonus&rdquo; blog post, because I&rsquo;m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.</p> +<!-- more--> + +<h2 id="lexical-scope-and-statistical-computation">Lexical scope and statistical computation</h2> + +<p>In <em>Lexical Scope and Statistical Computation</em>,<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">1</a></sup> Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.</p> + +<p>For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:</p> + +<ul> + <li><em>trivial</em>: no free variables allowed</li> + <li><em>static</em>: a free variable takes its value from a set of global variables</li> + <li><em>lexical</em>: a free variable takes the value of the binding that was in effect when the function was defined</li> + <li><em>dynamic</em>: a free variable takes the value of the most recent assignment to that variable</li></ul> + +<p>Note that under this set of definitions, <em>static scoping</em> is a separate scoping rule and not another name for <em>lexical scoping</em>.</p> + +<p>It is possible to simulate each of strategies in R. For fun, we can even construct &ldquo;factories&rdquo; that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)</p> + +<h3 id="template">Template</h3> + +<p>Our examples will follow the template given below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">factory</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="o">&lt;???&gt;</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">factory</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># error, 0, 1, or 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We want to define a <code>factory</code> that takes a function literal and returns a closure that implements the desired scoping rule.</p> + +<p>Our example consists of three definitions of <code>x</code>. On line 5, we assign <code>0</code> to <code>x</code> at the top level. On line 7, we assign <code>1</code> to <code>x</code> inside function <code>h</code>, where we also create the closure. On line 12, we assign <code>2</code> to <code>x</code> inside the function <code>f</code> and right before we call <code>g</code>, which is the closure.</p> + +<p>Finally, we call <code>f</code> and observe the result:</p> + +<ul> + <li>Under trivial scoping, no free variables are allowed, so <code>f()</code> should result in an error.</li> + <li>Under static scoping, free variables may only refer to global variables, so <code>f()</code> should return <code>0</code>.</li> + <li>Under lexical scoping, free variables refer to the variables in scope when the function was defined, so <code>f()</code> should return <code>1</code>.</li> + <li>Under dynamic scoping, free variables take the value from the most recent assignment, so <code>f()</code> should return <code>2</code>.</li></ul> + +<p>We will implement the body of <code>factory</code> in only 3&ndash;5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to <code>factory</code> on line 10.</p> + +<h3 id="trivial-scoping">Trivial scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeTrivial</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">baseenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeTrivial</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># Error in f(0) : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>substitute</code> returns the unevaluated parse tree for <code>fun</code>. In other words, it obtains the literal argument that was passed for <code>fun</code>. This works because of call-by-need semantics in R: function arguments are packaged up into <em>promises</em>. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">2</a></sup> discusses laziness in R in more detail.</p> + +<p>In this example, on line 8, we call <code>factory</code> with <code>function(a) x+a</code> as the argument for the formal parameter <code>fun</code>. Then, we evaluate that parse tree with <code>eval</code>.</p> + +<p>At this point, <code>res</code> is the closure with expression <code>function(a) x+a</code> and a reference to the environment of <code>makeTrivial</code>. On line 3, we change that reference to <code>baseenv()</code>, which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.</p> + +<p>Therefore, variable lookup in the function literal will only search the base environment, so <code>f()</code> results in an error.</p> + +<h3 id="static-scoping">Static scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeStatic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">globalenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeStatic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, on line 3, we update the environment of <code>res</code> to refer to <code>globalenv()</code>, which is the top-level environment where globals are defined.</p> + +<p>Therefore, variable lookup searches the top-level environment, so <code>f()</code> returns <code>0</code>.</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeLexical</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeLexical</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of <code>res</code> to <code>parent.frame()</code>, which is the environment of the function (<code>h</code>) that called the current function (<code>makeLexical</code>). This allows us to simulate lexical scoping, as if the function literal was evaluated inside <code>h</code>, rather than <code>makeLexical</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>h</code>, so <code>f()</code> returns <code>1</code>.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeDynamic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">function</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="nf">res</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> + <span class="p">}</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeDynamic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, we need another level of indirection. <code>makeDynamic</code> returns an anonymous function literal. The anonymous function takes <code>...</code>, which represents an arbitrary list of arguments, and then on line 5 we call <code>res</code> with those exact arguments. Note that we set the environment of <code>res</code> to be the environment of the <em>caller</em> of the anonymous function. Because of the multiple levels of indirection, the caller is <code>f</code>, on line 17.</p> + +<p>On line 12, <code>makeDynamic</code> returns a closure for the anonymous function. <code>h</code> returns that closure when it is called, and assigns it to <code>g</code>. When <code>g</code> is called on line 17, the function literal <code>function(a) x+a</code> is finally evaluated, and its environment is set to the environment of <code>f</code>, the caller of <code>g</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>f</code>, so <code>f()</code> returns <code>2</code>.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the <a href="/blog/2019/09/10/scoping-in-r/">previous post</a>, it&rsquo;s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.</p> + +<p>And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.</p> + +<p><em>I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>A. Goel and J. Vitek. &ldquo;On the Design, Implementation and Use of Laziness in R,&rdquo; in <em>Proceedings of the ACM in Programming Languages (PACMPL)</em>, vol. 3, no. OOPSLA, 2019. To appear. [<a href="http://janvitek.org/pubs/oopsla19a.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">↩</a></p></li></ol></div> + + Scoping in R + http://prl.ccs.neu.edu/blog/2019/09/10/scoping-in-r/?utm_source=r&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-scoping-in-r + Tue, 10 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">previous post</a> of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: <em>is R lexically or dynamically scoped?</em></p> +<!-- more--> + +<p>Recall the example program from before:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s examine what happens when we run this example. First, we create a mapping for <code>x</code> in the top-level environment. On line 2, we define a function <code>f</code>, which returns the value of some <code>x</code>. On line 3, we define a function <code>g</code>, which creates a new mapping for <code>x</code>, and then calls <code>f</code>. Note that the assignment on line 4 does <em>not</em> update the definition on line 1.</p> + +<p>When <code>f</code> is called, it needs to look up the value of <code>x</code>. In other words, does the reference of <code>x</code> on line 2 refer to the assignment on line 1 or the assignment on line 4? If <code>f</code> returns <code>1</code>, then the behaviour matches lexical scoping. If it returns <code>2</code>, then the behaviour matches dynamic scoping.</p> + +<p>When we run this example, the result is <code>1</code>. This implies that R is lexically scoped.</p> + +<p>But there&rsquo;s more to this story. In the rest of this blog post, I&rsquo;ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.</p> + +<p>The <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">next and final part</a> of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.</p> + +<h2 id="r-is-lexically-scoped-but">R is lexically scoped, but&hellip;</h2> + +<p>In <em>Evaluating the Design of the R Language</em>,<sup><a href="#2019-09-10-scoping-in-r-footnote-1-definition" name="2019-09-10-scoping-in-r-footnote-1-return">1</a></sup> Morandat, Hill, Osvald, and Vitek write:</p> + +<blockquote> + <p>As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.</p></blockquote> + +<p>In other words, R provides many different &ldquo;escape hatches&rdquo;&mdash;ways to bypass lexical scoping. Additionally, even without escape hatches, some of R&rsquo;s functionality can be surprising.</p> + +<h3 id="functions-environments-and-variables-in-r">Functions, environments, and variables in R</h3> + +<p>Before we look at some examples, I think it&rsquo;s useful to briefly discuss some of the core concepts in R that relate to scoping.</p> + +<ul> + <li> + <p><strong>Functions.</strong> R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.</p></li> + <li> + <p><strong>Environments.</strong> An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the &ldquo;enclosing&rdquo; environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.</p></li> + <li> + <p><strong>Variable lookup.</strong> When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.</p></li> + <li> + <p><strong>Variable assignment.</strong> <code>&lt;-</code> is the variable assignment operator in R. The expression <code>x &lt;- 1</code> assigns the value <code>1</code> to the variable <code>x</code> in the current environment. If a mapping for <code>x</code> already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.</p></li></ul> + +<p>From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.<sup><a href="#2019-09-10-scoping-in-r-footnote-2-definition" name="2019-09-10-scoping-in-r-footnote-2-return">2</a></sup></p> + +<p>On the other hand, variable lookup depends on the run-time state of the program&mdash;names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.</p> + +<p>The following examples will make this clear.</p> + +<h3 id="examples">Examples</h3> + +<h4 id="adding-variable-mappings-at-run-time">Adding variable mappings at run time</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it creates a function <code>g</code> that returns <code>x</code>, assigns <code>2</code> to <code>x</code>, and then calls <code>g</code>. When <code>g</code> is called, it looks up <code>x</code>. Since no mapping is found in <code>g</code>&rsquo;s environment, it searches in the enclosing environment (<code>f</code>&rsquo;s), and finds that <code>x</code> has value <code>2</code>. Therefore, <code>g</code> returns <code>2</code>.</p> + +<p>Note that the <code>x</code> on line 3 is resolved only when function <code>g</code> is called, not when it is defined. However, when <code>g</code> is defined, its environment has a reference to <code>f</code>&rsquo;s environment. Therefore, as long as <code>x</code> is defined <em>before</em> <code>g</code> is called, the lookup will always succeed.</p> + +<p>Here&rsquo;s a second example:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>f</code> is a function that branches on its argument, <code>b</code>. If <code>b</code> evaluates to true, then the expression <code>x &lt;- 2</code> is evaluated, and a mapping for <code>x</code> is created in <code>f</code>&rsquo;s environment. Otherwise, no mapping is created.</p> + +<p>When we look up the value of <code>x</code> on line 5, R will first search the function&rsquo;s environment. If <code>b</code> evaluated to true, then R will find a value for <code>x</code>, which is <code>2</code>. Otherwise, R will search in the enclosing environment of <code>f</code>, and find that <code>x</code> is <code>1</code>.</p> + +<p>Both of these examples vaguely resemble dynamic scoping, in that <code>x</code> takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.</p> + +<h4 id="function-lookup">Function lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> <span class="c1"># not an error</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">(</span><span class="m">42</span><span class="p">)</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.</p> + +<p>In this example, we call <code>g</code> with the argument <code>42</code>, which is not a function. Then, in the body of <code>g</code>, we call <code>f(0)</code> on line 3, which requires looking up <code>f</code>. Although there is an <code>f</code> in the environment of <code>g</code>, its value is <code>42</code>, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so <code>f(0)</code> returns <code>0</code>.</p> + +<p>This behaviour exists because <code>c</code> is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.</p> + +<h4 id="super-assignment">Super assignment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="n">x</span> <span class="o">&lt;&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +<span class="n">x</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>&lt;&lt;-</code> is the &ldquo;super assignment&rdquo; operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.</p> + +<p>In the above program, we define <code>x</code> to be <code>0</code> at the top level, and then define the function <code>f</code>. When we call <code>f</code> on line 7, it assigns <code>1</code> to <code>x</code> on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, <code>f</code> returns <code>x</code>, which is looked up from the local environment and has value <code>1</code>. Finally, line 8 looks up <code>x</code> from the top level environment, which has value <code>2</code>.</p> + +<h4 id="evaluating-arbitrary-code">Evaluating arbitrary code</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">eval</span><span class="p">(</span><span class="nf">parse</span><span class="p">(</span><span class="n">text</span> <span class="o">=</span> <span class="n">t</span><span class="p">))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="s">"x &lt;- 0"</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument <code>t</code>, which happens to be the string <code>"x &lt;- 0"</code>. Then, when line 4 executes, the lookup of <code>x</code> returns <code>0</code>.</p> + +<h4 id="simulating-dynamic-scope">Simulating dynamic scope</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.frame</span><span class="p">())</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 3, we perform an explicit variable lookup for <code>x</code>, but we do so in the environment <code>parent.frame()</code>, which refers to the calling function&rsquo;s environment, in this case, <code>g</code>&rsquo;s environment.. Therefore, the lookup returns <code>2</code>.</p> + +<p>Note that R has a similarly named function, <code>parent.env(e)</code> which returns the <em>enclosing environment</em> of the given environment <code>e</code>.</p> + +<h4 id="constructing-an-arbitrary-environment">Constructing an arbitrary environment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">e</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> + <span class="n">e</span><span class="o">$</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">3</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="n">e</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 3</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it constructs a new environment, <code>e</code>, which is initially empty. (By default, its enclosing environment is the current environment, which is <code>f</code>&rsquo;s.) Next, on line 4, it directly adds a mapping to that environment, assigning <code>3</code> to <code>x</code>. Then, on line 5, the lookup is explicitly done in environment <code>e</code>, so <code>f</code> returns <code>3</code>.</p> + +<h4 id="deleting-mappings">Deleting mappings</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">rm</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.env</span><span class="p">(</span><span class="nf">environment</span><span class="p">()))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># Error in f() : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Not only is it possible to dynamically add and modify mappings in R, but it is also possible to <em>delete</em> mappings. This is what line 3 does: it explicitly removes the mapping for <code>x</code> from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when <code>f</code> is called, the lookup of <code>x</code> fails and an error is raised.</p> + +<h4 id="infinite-loop-during-variable-lookup">Infinite loop during variable lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">enva</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="n">envb</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">enva</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">envb</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">envb</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="nf">environment</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="nf">f</span><span class="p">()</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.</p> + +<p>On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: <code>enva</code>&rsquo;s enclosing environment is <code>envb</code>, and <code>envb</code>&rsquo;s enclosing environment is <code>enva</code>.</p> + +<p>On line 5, we define a function with a free variable, <code>x</code>, but on line 6, we set <code>f</code>&rsquo;s environment to be <code>enva</code>. Finally, we call <code>f</code>.</p> + +<p>When the body of <code>f</code> is evaluated, it needs to look up <code>x</code>. Lookup starts in <code>f</code>&rsquo;s environment, which we set to be <code>enva</code>. Since no mapping for <code>x</code> is found, lookup continues in <code>enva</code>&rsquo;s enclosing environment, which is <code>envb</code>. However, <code>envb</code> is also empty, so lookup continues in its enclosing environment, which is <code>enva</code>, and now lookup results in an infinite loop.</p> + +<h3 id="an-intuition-for-scoping-in-r">An intuition for scoping in R</h3> + +<p>Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1"># example 1</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> + +<span class="c1"># example 2</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>It seems that <code>x</code> takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What&rsquo;s missing from our intuition?</p> + +<p>The key insight is that R is <em>function scoped</em>. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <code>&lt;-</code> <em>does not create a new scope</em>, and it is more useful to think of it as a mutation <em>on the current environment</em>. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)</p> + +<p>In a sense, it might be more accurate to say that R <em>environments</em> are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.</p> + +<h2 id="discussion">Discussion</h2> + +<p>All of this might make you a little uncomfortable, and uncertain about R&rsquo;s scoping rules.</p> + +<p>On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply &ldquo;lexical-like&rdquo; behaviour, and the creators have confirmed that lexical scoping was the intent.</p> + +<p>On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment&mdash;but this is not consistent with R&rsquo;s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>This ambiguity shows up in a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-3-definition" name="2019-09-10-scoping-in-r-footnote-3-return">3</a></sup> where the authors write:</p> + +<blockquote> + <p>Furthermore, because variable scoping in R is dynamic and can be modified at the language level [&hellip;] it cannot be trivially guaranteed that <code>x</code> is going to point to the same data structure throughout the entire execution of the loop.</p></blockquote> + +<p>It is true that a variable <code>x</code> may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.</p> + +<p>It is true that variable <em>lookup</em> is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of <em>dynamic scope</em>, then it would be fair to say that R is dynamically scoped.</p> + +<p>But if your definition of <em>dynamic scope</em> is &ldquo;a variable is bound to the most recent assignment during the program&rsquo;s execution,&rdquo; then it is not correct to say R is dynamically scoped.</p> + +<p>I think we have this ambiguity because <em>scope</em> (the places in a program where a variable can be referenced) and <em>variable lookup</em> or <em>name resolution</em> (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.</p> + +<p>Personally, I prefer the definition of <em>scope</em> that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is <em>both</em> lexically and dynamically scoped, or that R is <em>neither</em> lexically and dynamically scoped.</p> + +<p>I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language&mdash;when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.</p> + +<p>Regardless, it is very easy to miscommunicate, so I think it&rsquo;s important to be very clear and make sure you and your audience know what definitions of scoping you&rsquo;re using!</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>This entire adventure started when we were working on a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-4-definition" name="2019-09-10-scoping-in-r-footnote-4-return">4</a></sup> and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!</p> + +<p>This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.</p> + +<p>To summarize, I would say that <em>scope</em> refers to the places in a program where a variable is visible and can be referenced. Under <em>lexical scoping</em>, the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program. Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment during the program&rsquo;s execution.</p> + +<p>I would say that R <em>aims</em> to be lexically scoped&mdash;it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!</p> + +<p>If you want more content about R and scoping, the <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">third and final part</a> of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.</p> + +<p><strong>Edited 2020/02/21:</strong> For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), <a href="http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/">this blog post</a> has some nice examples and diagrams.</p> + +<p><em>I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.</em></p> + +<p><em>If you liked this post, you may also be interested in the following Twitter threads about R: <a href="https://twitter.com/mhyee/status/1063983175163158531">one</a>, <a href="https://twitter.com/mhyee/status/1067818720532316166">two</a> and <a href="https://twitter.com/mhyee/status/1074744049951739905">three</a>.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>F. Morandat, B. Hill, L. Osvald, J. Vitek. &ldquo;Evaluating the Design of the R Language,&rdquo; in <em>Proceedings of the European Conference on Object-Oriented Programming (ECOOP)</em>, 2012. [<a href="https://doi.org/10.1007/978-3-642-31057-7_6">DOI</a>][<a href="http://janvitek.org/pubs/ecoop12.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. &ldquo;Lexical Scope and Statistical Computing&rdquo;, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-2-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-3-definition" class="footnote-definition"> + <p>L. Stadler, A. Welc, C. Humer, and M. Jordan. &ldquo;Optimizing R Language Execution via Aggressive Speculation,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2016. [<a href="https://doi.org/10.1145/2989225.2989236">DOI</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-3-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-4-definition" class="footnote-definition"> + <p>O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. &ldquo;R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2019. To appear. [<a href="http://janvitek.org/pubs/dls19.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-4-return">↩</a></p></li></ol></div> \ No newline at end of file diff --git a/blog/feeds/racket.atom.xml b/blog/feeds/racket.atom.xml new file mode 100644 index 00000000..ef540298 --- /dev/null +++ b/blog/feeds/racket.atom.xml @@ -0,0 +1,486 @@ + + + PRL Blog: Posts tagged 'racket' + + + urn:http-prl-ccs-neu-edu:-blog-tags-racket-html + 2017-05-26T17:00:28Z + + Racket 6.9 and Windows 10 Creators Update + + urn:http-prl-ccs-neu-edu:-blog-2017-05-26-racket-6-9-and-windows-10-creators-update + 2017-05-26T17:00:28Z + 2017-05-26T17:00:28Z + + Leif Andersen + +<p><a href="http://racket-lang.org/">Racket</a> 6.9 was released in April and it has been smooth sailing for many people. However, some people using the <a href="https://blogs.windows.com/windowsexperience/2017/04/11/whats-new-in-the-windows-10-creators-update/">Windows 10 Creators Update</a> have been experiencing <a href="https://github.com/racket/racket/issues/1671">crashes</a>, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.</p> + +<p>The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.</p> + +<p>A VirtualBox image with Racket preinstalled can be downloaded here:</p> + +<ul> + <li><a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova</a></li></ul> + +<p>The username and password for this machine are both <code>racket</code>.</p> +<!-- more--> + +<ol> + <li> + <p>The first thing you need to install is virtualization software. In principle it doesn&rsquo;t matter what you install, but for this tutorial, we will use <a href="https://www.virtualbox.org/">VirtualBox</a>. Go to their <a href="https://www.virtualbox.org/wiki/Downloads">downloads</a> page and download and install the version for your platform (most likely Windows).</p></li> + <li> + <p>Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which <a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">you can download here</a>. The rest of this tutorial will assume you are using this image.</p></li> + <li> + <p>Start up VirtualBox and import the virtual machine. You can do this by clicking on <code>File -&gt; Import Appliance</code>. In the dialog, select the image you downloaded and hit <code>Continue</code>. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click <code>Import</code>.</p></li> + <li> + <p>After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit <code>Start</code>. Once started up, you will see DrRacket and Firefox on the VM&rsquo;s desktop.</p></li></ol> + + Measuring GC latencies in Haskell, OCaml, Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + 2016-05-24T10:51:34Z + 2016-05-24T10:51:34Z + + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/racket.rss.xml b/blog/feeds/racket.rss.xml new file mode 100644 index 00000000..16672945 --- /dev/null +++ b/blog/feeds/racket.rss.xml @@ -0,0 +1,484 @@ + + + + PRL Blog: Posts tagged 'racket' + PRL Blog: Posts tagged 'racket' + http://prl.ccs.neu.edu/blog/tags/racket.html + Fri, 26 May 2017 17:00:28 UT + Fri, 26 May 2017 17:00:28 UT + 1800 + + Racket 6.9 and Windows 10 Creators Update + http://prl.ccs.neu.edu/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/?utm_source=racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-26-racket-6-9-and-windows-10-creators-update + Fri, 26 May 2017 17:00:28 UT + Leif Andersen + +<p><a href="http://racket-lang.org/">Racket</a> 6.9 was released in April and it has been smooth sailing for many people. However, some people using the <a href="https://blogs.windows.com/windowsexperience/2017/04/11/whats-new-in-the-windows-10-creators-update/">Windows 10 Creators Update</a> have been experiencing <a href="https://github.com/racket/racket/issues/1671">crashes</a>, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.</p> + +<p>The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.</p> + +<p>A VirtualBox image with Racket preinstalled can be downloaded here:</p> + +<ul> + <li><a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova</a></li></ul> + +<p>The username and password for this machine are both <code>racket</code>.</p> +<!-- more--> + +<ol> + <li> + <p>The first thing you need to install is virtualization software. In principle it doesn&rsquo;t matter what you install, but for this tutorial, we will use <a href="https://www.virtualbox.org/">VirtualBox</a>. Go to their <a href="https://www.virtualbox.org/wiki/Downloads">downloads</a> page and download and install the version for your platform (most likely Windows).</p></li> + <li> + <p>Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which <a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">you can download here</a>. The rest of this tutorial will assume you are using this image.</p></li> + <li> + <p>Start up VirtualBox and import the virtual machine. You can do this by clicking on <code>File -&gt; Import Appliance</code>. In the dialog, select the image you downloaded and hit <code>Continue</code>. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click <code>Import</code>.</p></li> + <li> + <p>After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit <code>Start</code>. Once started up, you will see DrRacket and Firefox on the VM&rsquo;s desktop.</p></li></ol> + + Measuring GC latencies in Haskell, OCaml, Racket + http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/?utm_source=racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-05-24-measuring-gc-latencies-in-haskell-ocaml-racket + Tue, 24 May 2016 10:51:34 UT + Gabriel Scherer + +<p>James Fisher has a blog post on a case where GHC&rsquo;s runtime system imposed unpleasant latencies on their Haskell program:</p> + +<blockquote> + <p><a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">Low latency, large working set, and GHC&rsquo;s garbage collector: pick two of three</a></p></blockquote> + +<p>The blog post proposes a very simple, synthetic benchmark that exhibits the issue &mdash; basically, latencies incurred by copy time &mdash; with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.</p> + +<p>Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&amp;sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.</p> + +<p>The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a &ldquo;ramp-up&rdquo; effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.</p> + +<p>Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC&rsquo;s garbage collector. This does not mean that GHC&rsquo;s latencies are bad in general, or that the other tested languages have smaller latencies in general.</p> + +<p>The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:</p> + +<ul> + <li>git: <a href="https://gitlab.com/gasche/gc-latency-experiment.git">https://gitlab.com/gasche/gc-latency-experiment.git</a></li> + <li>files: <a href="https://gitlab.com/gasche/gc-latency-experiment/tree/master">https://gitlab.com/gasche/gc-latency-experiment/tree/master</a></li></ul> +<!-- more--> + +<h2 id="the-haskell-benchmark">The Haskell benchmark</h2> + +<p>James Fisher&rsquo;s Haskell benchmark is very simple: it creates an association table in which medium-size strings are inserted repeatedly &mdash; a million times. When the channel reaches 200_000 messages, a string is deleted each time a string is created, to keep the total working size constant.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Exception</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Exception</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Control.Monad</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Monad</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.ByteString</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ByteString</span><span class="w"></span> +<span class="kr">import</span><span class="w"> </span><span class="k">qualified</span><span class="w"> </span><span class="nn">Data.Map.Strict</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Map</span><span class="w"></span> + +<span class="kr">data</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="o">!</span><span class="kt">Int</span><span class="w"> </span><span class="o">!</span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">))</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="p">(</span><span class="kt">Msg</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="p">)</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">msgId</span><span class="w"> </span><span class="n">msgContent</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="mi">200000</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">size</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">deleteMin</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="o">..</span><span class="mi">1000000</span><span class="p">])</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>To compile and run the program (<code>make run-haskell</code> also works in my repository):</p> + +<pre><code>ghc -O2 -optc-O3 Main.hs # compile the program +./Main +RTS -s # run the program (with GC instrumentation enabled)</code></pre> + +<p>On my machine, running the program takes around 1.5s. We are not interested in the total running time (the <em>throughput</em> of the algorithm), but in the pause times induced by the GC: the worst pause time is 51ms (milliseconds), which is the same as the one reported by the blog post &mdash; and there it is considered excessive, with an expected worst-case latency of at most &ldquo;a few milliseconds&rdquo;.</p> + +<p>(I did my testing with GHC 7.8, Fischer reports results with 7.10, they are essentially the same.)</p> + +<p>This Haskell code makes two assumption about the <code>Map</code> data structure (immutable associative maps) that make the benchmark more cumbersome to port to other languages. It assumes that the element count is pre-cached in the data structure and thus <code>Map.size</code> is constant-time &mdash; for both OCaml and Racket it is linear. It also uses a key ordering that makes it easy to remove the smallest key &mdash; OCaml does this as well, but Racket uses hashes instead.</p> + +<p>I initially worked around this by storing count and minimum-key information in the ported versions, but in fact it&rsquo;s much nicer to write a variant of the benchmark, with the same behavior, that does not require these specific features:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kr">type</span><span class="w"> </span><span class="kt">Msg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="kt">ByteString</span><span class="w"></span> +<span class="kr">type</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="kt">Map</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> + +<span class="nf">windowSize</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">200000</span><span class="w"></span> +<span class="nf">msgCount</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="mi">1000000</span><span class="w"></span> + +<span class="nf">message</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Msg</span><span class="w"></span> +<span class="nf">message</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">ByteString</span><span class="o">.</span><span class="n">replicate</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="p">(</span><span class="n">fromIntegral</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"></span> + +<span class="nf">pushMsg</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Chan</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">Int</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="kt">Chan</span><span class="w"></span> +<span class="nf">pushMsg</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="ow">=</span><span class="w"></span> +<span class="w"> </span><span class="kt">Exception</span><span class="o">.</span><span class="n">evaluate</span><span class="w"> </span><span class="o">$</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">windowSize</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">inserted</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">insert</span><span class="w"> </span><span class="n">highId</span><span class="w"> </span><span class="p">(</span><span class="n">message</span><span class="w"> </span><span class="n">highId</span><span class="p">)</span><span class="w"> </span><span class="n">chan</span><span class="w"> </span><span class="kr">in</span><span class="w"></span> +<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> +<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">delete</span><span class="w"> </span><span class="n">lowId</span><span class="w"> </span><span class="n">inserted</span><span class="w"></span> + +<span class="nf">main</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">IO</span><span class="w"> </span><span class="nb">()</span><span class="w"></span> +<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Monad</span><span class="o">.</span><span class="n">foldM_</span><span class="w"> </span><span class="n">pushMsg</span><span class="w"> </span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="w"> </span><span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="n">msgCount</span><span class="p">]</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This variant has the same running times and worst-case pause, 50ms, as the original program.</p> + +<h3 id="explaining-haskell-results">Explaining Haskell results</h3> + +<p>James Fischer explains that the reason why the latencies are this high (50ms is considered high) is that while GHC&rsquo;s garbage collector is generational, its older generation still uses a stop-and-copy scheme. This means that when it contains lots of large objects, a lot of time is spent copying them.</p> + +<p>The <a href="https://blog.pusher.com/latency-working-set-ghc-gc-pick-two/">original blog post</a> contains a more detailed description of the problem and of various optimizations that may be attempted. Unfortunately, it seems that it is currently impossible to optimize that kind of workloads by tuning the code or GC parameters: the copying behavior of the old heap cannot really be worked-around currently.</p> + +<p>As a meta-comment, one possible explanation for why this design choice was made might be that a lot of effort was invested in the Haskell&rsquo;s GC to support concurrent mutators (a multi-core runtime). The additional complexity imposed by this extremely challenging and useful requirement may have encouraged runtime authors to keep the general GC architecture as simple as reasonably possible, which could explain this choice of using the same collection strategy in all generational spaces.</p> + +<h2 id="ocaml-version">OCaml version</h2> + +<p>The code can easily be ported into OCaml, for example as follows:</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">open</span> <span class="nc">Batteries</span> +<span class="k">module</span> <span class="nc">IMap</span> <span class="o">=</span> <span class="nn">Map</span><span class="p">.</span><span class="nc">Make</span><span class="o">(</span><span class="nc">Int</span><span class="o">)</span> + +<span class="k">let</span> <span class="n">message</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">String</span><span class="p">.</span><span class="n">make</span> <span class="mi">1024</span> <span class="o">(</span><span class="nn">Char</span><span class="p">.</span><span class="n">chr</span> <span class="o">(</span><span class="n">n</span> <span class="ow">mod</span> <span class="mi">256</span><span class="o">))</span> + +<span class="k">let</span> <span class="n">window_size</span> <span class="o">=</span> <span class="mi">200_000</span> +<span class="k">let</span> <span class="n">msg_count</span> <span class="o">=</span> <span class="mi">1_000_000</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> + <span class="nn">Seq</span><span class="p">.</span><span class="n">init</span> <span class="n">msg_count</span> <span class="o">(</span><span class="k">fun</span> <span class="n">i</span> <span class="o">-&gt;</span> <span class="n">i</span><span class="o">)</span> + <span class="o">|&gt;</span> <span class="nn">Seq</span><span class="p">.</span><span class="n">fold_left</span> <span class="n">push_msg</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">empty</span> <span class="o">|&gt;</span> <span class="n">ignore</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Evaluating throughput is not the point, and the balanced maps used by the Haskell and OCaml are certainly implemented in slightly different ways that would explain any performance difference, but I was still amused to see the total runtime be essentially the same: 1.5s.</p> + +<p>To measure the maximal pause time, there are two options:</p> + +<ul> + <li> + <p>use the new instrumented runtime contributed by Damien Doligez in OCaml 4.03; this works but, being a relatively new feature with not much usability effort put into it, it&rsquo;s far from being as convenient as GHC&rsquo;s <code>+RTS -s</code> parameter.</p></li> + <li> + <p>Simply measure the time spend in each iteration (pushing a message), and using this as an upper bound on the pause time: clearly any GC pause cannot pause for more time than the iteration takes. (With my Makefile, <code>make run-ocaml</code>)</p></li></ul> + +<p>To use the new instrumented runtime, you need to have an OCaml compiler, version 4.03.0, compiled with the <code>--with-instrumented-runtime</code> configure-time switch. Then, you can use the <code>i</code>-variant (<code>i</code> for &ldquo;instrumented&rdquo;) of the runtime that is compiled with instrumentation enabled. (My makefile rule <code>make +run-ocaml-instrumented</code> does this for you, but you still need a switch compiled with the instrumented runtime.)</p> + +<pre><code>ocamlbuild -tag "runtime_variant(i)" main.native +OCAML_INSTR_LOG=ocaml.log ./main.native</code></pre> + +<p>The log file <code>ocaml.log</code> will then contain a low-level log of all GC-related runtime calls, with nanosecond time, in a format made for machine rather than human consumption. The tools <code>ocaml-instr-report</code> and <code>ocaml-instr-graph</code> of the OCaml source distribution (not installed by default, you need a source checkout), will parse them and display tables or graph. The entry point of interest for worst-case latency is <code>dispatch</code>, which contains the time spent in all GC activity. The relevant section of <code>ocaml-instr-report</code>&rsquo;s output shows:</p> + +<pre><code>==== dispatch: 2506 +470ns..1.0us: 1 (768ns) 0.04% +1.0us..2.2us: # 2 0.12% +2.2us..4.7us: ### 8 0.44% +4.7us..10us : #### 10 0.84% + 10us..22us : 1 (14us) 0.88% + 22us..47us : 0.88% + 47us..100us: 0.88% +100us..220us: ## 3 1.00% +220us..470us: ########## 668 27.65% +470us..1.0ms: ########### 1795 99.28% +1.0ms..2.2ms: ##### 17 99.96% +2.2ms..4.7ms: 1 (2.7ms) 100.00%</code></pre> + +<p>As you can see, most pauses are between 220µs and 1ms, with the longest pause being 2.7ms.</p> + +<p>The other approach to measure latency for this program, which works on older OCaml versions without an instrumented runtime, is just to insert explicit timing calls and compute the worst-case time of an iteration &mdash; as an over-approximation over the max pause time, assuming that the actual insertion/deletion time is small.</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="k">let</span> <span class="n">worst</span> <span class="o">=</span> <span class="n">ref</span> <span class="mi">0</span><span class="o">.</span> +<span class="k">let</span> <span class="n">time</span> <span class="n">f</span> <span class="o">=</span> + <span class="k">let</span> <span class="n">before</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">after</span> <span class="o">=</span> <span class="nn">Unix</span><span class="p">.</span><span class="n">gettimeofday</span> <span class="bp">()</span> <span class="k">in</span> + <span class="n">worst</span> <span class="o">:=</span> <span class="n">max</span> <span class="o">!</span><span class="n">worst</span> <span class="o">(</span><span class="n">after</span> <span class="o">-.</span> <span class="n">before</span><span class="o">);</span> + <span class="n">result</span> + +<span class="k">let</span> <span class="n">push_msg</span> <span class="n">chan</span> <span class="n">high_id</span> <span class="o">=</span> <span class="n">time</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-&gt;</span> + <span class="k">let</span> <span class="n">low_id</span> <span class="o">=</span> <span class="n">high_id</span> <span class="o">-</span> <span class="n">window_size</span> <span class="k">in</span> + <span class="k">let</span> <span class="n">inserted</span> <span class="o">=</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">add</span> <span class="n">high_id</span> <span class="o">(</span><span class="n">message</span> <span class="n">high_id</span><span class="o">)</span> <span class="n">chan</span> <span class="k">in</span> + <span class="k">if</span> <span class="n">low_id</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">inserted</span> + <span class="k">else</span> <span class="nn">IMap</span><span class="p">.</span><span class="n">remove</span> <span class="n">low_id</span> <span class="n">inserted</span> + +<span class="c">(* ..main loop.. *)</span> +<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Printf</span><span class="p">.</span><span class="n">printf</span> <span class="s2">"Worst pause: %.2E</span><span class="se">\n</span><span class="s2">"</span> <span class="o">!</span><span class="n">worst</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this version reports a worst-case latency of 2ms seconds on my machine (I use the <code>%E</code> formatter for scientific notation, so it gets printed as <code>2.03E-03</code>), which is in line with the instrumented runtime &mdash; actually slightly lower, as the instrumentation may add some overhead.</p> + +<p>A downside of this poor man worst-latency computation approach is that we only get the worst time, not any kind of timing distribution.</p> + +<h3 id="explaining-ocaml-results">Explaining OCaml results</h3> + +<p>The OCaml GC has had reliable incremental phases implemented by default for a long time, and does not use a copying strategy for its old generation. It is mark&amp;sweep, executed well, so it was predictable from the start that this specific benchmark would not be a worst-case for OCaml.</p> + +<p>The latest released OCaml version, OCaml 4.03.0, has seen work by Damien Doligez to improve the worst-case latency in some situations, motivated by the industrial use-cases of Jane Street. In particular, the latency <em>instrumentation</em> tools that I&rsquo;m using above were developed by Damien on this occasion. I checked with the second measurement strategy that the latency is just as good on previous OCaml versions: this particular use-case was not in need of improvement before 4.03.</p> + +<h2 id="racket-version">Racket version</h2> + +<p>Max New wrote a first version of Racket port of this benchmark &mdash; he had to explicitly keep track of the map count and minimum key to match the original GHC version. I adapted his code to my simplified variant, and it looks rather similar to the other implementations.</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">#</span><span class="nv">lang</span> <span class="nv">racket/base</span> +<span class="p">(</span><span class="nf">require</span> <span class="nv">racket/match</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">window-size</span> <span class="mi">200000</span><span class="p">)</span> +<span class="p">(</span><span class="k">define </span><span class="nv">msg-count</span> <span class="mi">2000000</span><span class="p">)</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">message</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-bytes</span> <span class="mi">1024</span> <span class="p">(</span><span class="nb">modulo </span><span class="nv">n</span> <span class="mi">256</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">id-high</span><span class="p">)</span> + <span class="p">(</span><span class="k">define </span><span class="nv">id-low</span> <span class="p">(</span><span class="nf">id-high</span> <span class="o">.</span> <span class="nv">-</span> <span class="o">.</span> <span class="nv">window-size</span><span class="p">))</span> + <span class="p">(</span><span class="k">define </span><span class="nv">inserted</span> <span class="p">(</span><span class="nf">hash-set</span> <span class="nv">chan</span> <span class="nv">id-high</span> <span class="p">(</span><span class="nf">message</span> <span class="nv">id-high</span><span class="p">)))</span> + <span class="p">(</span><span class="k">if </span><span class="p">(</span><span class="nf">id-low</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">inserted</span> + <span class="p">(</span><span class="nf">hash-remove</span> <span class="nv">inserted</span> <span class="nv">id-low</span><span class="p">)))</span> + +<span class="p">(</span><span class="k">define </span><span class="nv">_</span> + <span class="p">(</span><span class="nf">for/fold</span> + <span class="p">([</span><span class="nv">chan</span> <span class="p">(</span><span class="nf">make-immutable-hash</span><span class="p">)])</span> + <span class="p">([</span><span class="nv">i</span> <span class="p">(</span><span class="nf">in-range</span> <span class="nv">msg-count</span><span class="p">)])</span> + <span class="p">(</span><span class="nf">push-msg</span> <span class="nv">chan</span> <span class="nv">i</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>I initially used the poor man approach of explicit timing calls to measure latency, but then switched to two better methods:</p> + +<ul> + <li> + <p>Sam Tobin-Hochstadt&rsquo;s <a href="https://github.com/samth/gcstats">gcstats</a> package makes Racket programs produce a summary of their runtime behavior in the same format as GHC&rsquo;s <code>+RTS -s</code> output, with in particular the worst-case pause time. It is also very easy to use:</p> + <pre><code>racket -l gcstats -t main.rkt</code></pre></li> + <li> + <p>By setting the environment variable <code>PLTSTDERR=debug@GC</code>, the racket runtime will log GC events on the standard error output. One can then grep for minor or major collections, or produce a histogram of running times through the following scripting soup I cooked myself:</p> + <pre><code>cat racket.log | grep -v total | cut -d' ' -f7 | sort -n | uniq --count</code></pre></li></ul> + +<p>Racket has an incremental GC that is currently experimental (it is not enabled by default as it can degrade throughput) and is enabled by setting the environment variable <code>PLT_INCREMENTAL_GC=1</code>. I compared with and without the incremental GC, and generally it shifts the latency histogram towards smaller latencies, but it turns out not to help so much for the worst-case latency without further tuning, for a reason I will explain. All results reported below use the incremental GC.</p> + +<p>On my machine, using the latest release Racket 6.5, the maximal pause time reported by <code>gcstats</code> is around 150ms, which is rather bad &mdash; the excessive pause of GHC was 50ms.</p> + +<h3 id="investigating-the-racket-results">Investigating the Racket results</h3> + +<p>I sent <a href="https://groups.google.com/forum/#!topic/racket-dev/AH6c-HGgzJ0">an email</a> to the racket-dev mailing list, hoping to get explanations and advice on how to improve the code to decrease GC latencies. (Remember that one problematic aspect of the GHC benchmark is that there is no real way for users to tweak the code to get better latencies for the same workflow. So we are evaluating default latencies but also tweakability.) It worked out quite well.</p> + +<p>First, Matthew Flatt immediately sent a few commits on the Racket codebase to improve some behaviors that were problematic on the benchmark. Using the development version of Racket instead of 6.5, the worst-case latency drops from 150ms to 120ms on my machine. All remaining times are reported using the development version.</p> + +<p>Matthew Flatt also analyzed the result and noticed that the worst-case latency systematically happens at the beginning of the benchmark, just after the channel reaches its maximal side of 200,000 messages. This is hard to see with the default benchmark parameters, where the &ldquo;ramp-up&rdquo; period of filling the channel takes one fifth of the total iterations. To see this clearly, I increased the iteration count from 1,000,000 to 10,000,000, then ran <code>make +run-racket-instrumented</code>. I can look at the pause time of major collections by doing <code>grep MAJ racket.log</code>, and on my machine I have:</p> + +<pre><code>GC: 0:MAJ @ 50,634K(+37,221K)[+1,560K]; free 5,075K(-5,075K) 12ms @ 373 +GC: 0:MAJ @ 101,983K(+35,024K)[+1,560K]; free 10,880K(-5,168K) 38ms @ 521 +GC: 0:MAJ @ 192,491K(+38,404K)[+1,560K]; free 8,174K(-24,030K) 56ms @ 810 +GC: 0:MAJ @ 377,716K(+49,259K)[+1,560K]; free 10,832K(-9,536K) 92ms @ 1571 +GC: 0:MAJ @ 742,630K(+59,881K)[+1,560K]; free 140,354K(-156,738K) 138ms @ 3321 +GC: 0:MAJ @ 1,214,486K(+112,313K)[+1,560K]; free 361,371K(-377,755K) 60ms @ 6046 +GC: 0:MAJ @ 1,417,749K(+138,410K)[+1,560K]; free 600,291K(-600,291K) 23ms @ 8553 +GC: 0:MAJ @ 1,400,780K(+155,379K)[+1,560K]; free 564,923K(-564,923K) 21ms @ 11048 +GC: 0:MAJ @ 1,408,812K(+147,347K)[+1,560K]; free 583,454K(-583,454K) 21ms @ 13506 +GC: 0:MAJ @ 1,404,757K(+151,402K)[+1,560K]; free 572,350K(-572,350K) 20ms @ 15983 +GC: 0:MAJ @ 1,407,842K(+148,317K)[+1,560K]; free 579,079K(-579,079K) 22ms @ 18438 +GC: 0:MAJ @ 1,405,641K(+150,518K)[+1,560K]; free 575,624K(-575,624K) 21ms @ 20907 +GC: 0:MAJ @ 1,405,833K(+150,326K)[+1,560K]; free 577,191K(-577,191K) 21ms @ 23362 +GC: 0:MAJ @ 1,405,763K(+150,396K)[+1,560K]; free 575,779K(-575,779K) 20ms @ 25897 +GC: 0:MAJ @ 1,406,444K(+149,715K)[+1,560K]; free 577,553K(-577,553K) 20ms @ 28348 +GC: 0:MAJ @ 1,406,409K(+149,750K)[+1,560K]; free 576,323K(-576,323K) 21ms @ 30827 +GC: 0:MAJ @ 1,407,054K(+149,105K)[+1,560K]; free 577,961K(-577,961K) 21ms @ 33290 +GC: 0:MAJ @ 1,404,903K(+151,256K)[+1,560K]; free 576,241K(-576,241K) 20ms @ 35774 +GC: 0:MAJ @ 1,406,551K(+149,608K)[+1,560K]; free 575,352K(-575,352K) 22ms @ 38251 +GC: 0:MAJ @ 1,405,775K(+150,384K)[+1,560K]; free 577,401K(-577,401K) 21ms @ 40730 +GC: 0:MAJ @ 1,406,015K(+150,144K)[+1,560K]; free 575,563K(-575,563K) 20ms @ 43254 +GC: 0:MAJ @ 1,406,129K(+150,030K)[+1,560K]; free 577,760K(-577,760K) 21ms @ 45730 +GC: 0:MAJ @ 1,406,157K(+150,002K)[+1,560K]; free 575,394K(-575,394K) 22ms @ 48220 +GC: 0:MAJ @ 1,406,514K(+149,645K)[+1,560K]; free 577,765K(-577,765K) 21ms @ 50697</code></pre> + +<p>Look at the evolution of major collection pause times: there is an early peek at <code>140ms</code>, but then pause times decrease and the steady state has sensibly shorter pauses of around <code>22ms</code>. By looking at the amount of memory freed during each collection, one can see that the peak corresponds to the first major collection that frees a lot of memory; it is the first major collection after the channel has reached its maximal size, and starts removing a lot of messages.</p> + +<p>My understanding of this behavior is that the incremental GC keeps some runtime parameter that observe the memory allocation patterns of the program, and try to predict when the next collection should be or how much work it should do. Matthew Flatt explains that this monitoring logic currently fails to adapt gracefully to the change of regime in our program, and incurs a large peak pause at this point.</p> + +<p>This is good news for our benchmark: sure, there is a very bad pause at the beginning of the program, but it&rsquo;s a one-time thing. It does not really affect the last decile of latency that is discussed in James Fischer&rsquo;s post, and would not be a problem during the steady state of an actual message-passing application.</p> + +<h3 id="tuning-the-racket-version">Tuning the Racket version</h3> + +<p>Matthew Flatt also remarked that by inserting explicit calls to the GC, one can get collection performed more often than Racket&rsquo;s heuristics demand and partly avoid the large peak pause. However, too frequent explicit collections hurt the program throughput.</p> + +<p>I experimented a bit and found that the peak pause issue could be partly mitigated by inserting explicit GC calls around the change of regime &mdash; around the iteration count that corresponds to the maximal channel size. I defined a function doing just that</p> + +<div class="brush: scheme"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k">define </span><span class="p">(</span><span class="nf">maybe-gc</span> <span class="nv">i</span><span class="p">)</span> + <span class="p">(</span><span class="nf">when</span> <span class="p">(</span><span class="k">and </span><span class="nv">gc-during-rampup</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&gt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">/</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nf">i</span> <span class="o">.</span> <span class="nv">&lt;</span> <span class="o">.</span> <span class="p">(</span><span class="nf">window-size</span> <span class="o">.</span> <span class="nv">*</span> <span class="o">.</span> <span class="mi">2</span><span class="p">))</span> + <span class="p">(</span><span class="nb">zero? </span><span class="p">(</span><span class="nb">modulo </span><span class="nv">i</span> <span class="mi">50</span><span class="p">)))</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;incremental</span><span class="p">)</span> + <span class="p">(</span><span class="nf">collect-garbage</span> <span class="ss">&#39;minor</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>which is controlled by a <code>gc-during-rampup</code> parameter that you can explicitly set to <code>#t</code> to experiment &mdash; explicit GC calls are disabled by default in my benchmark code. Then I just inserted a <code>(maybe-gc i)</code> call in the main loop.</p> + +<p>Because the extra GC calls happen only during rampup, the performance of the steady state are unchanged and the global cost on throughput is moderate (20% in my experiment with iteration count 2,000,000). This seems effective at mitigating the peak pause issue: the worst-case time on my machine is now only 38ms &mdash; the pauses during the steady state are unchanged, around 22ms.</p> + +<p>This is, of course, a hack; the long-term solution is to wait for Racket developers to devise better dynamic control strategies to avoid the ramp-up problem. Apparently, the incremental GC was previously tested on games that had simpler allocation profiles, such as short-lived memory allocations during each game tick, with no such a long ramp-up phase. But I was still interested in the fact that expert users can tweak the code to noticeably decrease the worst-case pause time.</p> + +<p>To summarize, Racket&rsquo;s incremental GC exhibits a decent-but-not-excellent steady state behavior, with maximal latencies of around 22ms, but currently suffers from a GC control issues that cause much larger pauses during the benchmark ramp-up period. Explicit GC calls can partly mitigate them.</p> \ No newline at end of file diff --git a/blog/feeds/reduction.atom.xml b/blog/feeds/reduction.atom.xml new file mode 100644 index 00000000..363a0658 --- /dev/null +++ b/blog/feeds/reduction.atom.xml @@ -0,0 +1,306 @@ + + + PRL Blog: Posts tagged 'reduction' + + + urn:http-prl-ccs-neu-edu:-blog-tags-reduction-html + 2016-11-02T21:10:18Z + + Beta Reduction (Part 1) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + 2016-11-02T21:10:18Z + 2016-11-02T21:10:18Z + + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/reduction.rss.xml b/blog/feeds/reduction.rss.xml new file mode 100644 index 00000000..9ff3d7cd --- /dev/null +++ b/blog/feeds/reduction.rss.xml @@ -0,0 +1,306 @@ + + + + PRL Blog: Posts tagged 'reduction' + PRL Blog: Posts tagged 'reduction' + http://prl.ccs.neu.edu/blog/tags/reduction.html + Wed, 02 Nov 2016 21:10:18 UT + Wed, 02 Nov 2016 21:10:18 UT + 1800 + + Beta Reduction (Part 1) + http://prl.ccs.neu.edu/blog/2016/11/02/beta-reduction-part-1/?utm_source=reduction&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + Wed, 02 Nov 2016 21:10:18 UT + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/scope.atom.xml b/blog/feeds/scope.atom.xml new file mode 100644 index 00000000..c32bd144 --- /dev/null +++ b/blog/feeds/scope.atom.xml @@ -0,0 +1,1008 @@ + + + PRL Blog: Posts tagged 'scope' + + + urn:http-prl-ccs-neu-edu:-blog-tags-scope-html + 2019-09-10T11:00:00Z + + Four Kinds of Scoping in R + + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-four-kinds-of-scoping-in-r + 2019-09-10T11:00:00Z + 2019-09-10T11:00:00Z + + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">first</a> and <a href="/blog/2019/09/10/scoping-in-r/">second</a> parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.</p> + +<p>In this third and final part of my blog series, I&rsquo;d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.</p> + +<p>This is a &ldquo;bonus&rdquo; blog post, because I&rsquo;m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.</p> +<!-- more--> + +<h2 id="lexical-scope-and-statistical-computation">Lexical scope and statistical computation</h2> + +<p>In <em>Lexical Scope and Statistical Computation</em>,<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">1</a></sup> Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.</p> + +<p>For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:</p> + +<ul> + <li><em>trivial</em>: no free variables allowed</li> + <li><em>static</em>: a free variable takes its value from a set of global variables</li> + <li><em>lexical</em>: a free variable takes the value of the binding that was in effect when the function was defined</li> + <li><em>dynamic</em>: a free variable takes the value of the most recent assignment to that variable</li></ul> + +<p>Note that under this set of definitions, <em>static scoping</em> is a separate scoping rule and not another name for <em>lexical scoping</em>.</p> + +<p>It is possible to simulate each of strategies in R. For fun, we can even construct &ldquo;factories&rdquo; that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)</p> + +<h3 id="template">Template</h3> + +<p>Our examples will follow the template given below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">factory</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="o">&lt;???&gt;</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">factory</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># error, 0, 1, or 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We want to define a <code>factory</code> that takes a function literal and returns a closure that implements the desired scoping rule.</p> + +<p>Our example consists of three definitions of <code>x</code>. On line 5, we assign <code>0</code> to <code>x</code> at the top level. On line 7, we assign <code>1</code> to <code>x</code> inside function <code>h</code>, where we also create the closure. On line 12, we assign <code>2</code> to <code>x</code> inside the function <code>f</code> and right before we call <code>g</code>, which is the closure.</p> + +<p>Finally, we call <code>f</code> and observe the result:</p> + +<ul> + <li>Under trivial scoping, no free variables are allowed, so <code>f()</code> should result in an error.</li> + <li>Under static scoping, free variables may only refer to global variables, so <code>f()</code> should return <code>0</code>.</li> + <li>Under lexical scoping, free variables refer to the variables in scope when the function was defined, so <code>f()</code> should return <code>1</code>.</li> + <li>Under dynamic scoping, free variables take the value from the most recent assignment, so <code>f()</code> should return <code>2</code>.</li></ul> + +<p>We will implement the body of <code>factory</code> in only 3&ndash;5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to <code>factory</code> on line 10.</p> + +<h3 id="trivial-scoping">Trivial scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeTrivial</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">baseenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeTrivial</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># Error in f(0) : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>substitute</code> returns the unevaluated parse tree for <code>fun</code>. In other words, it obtains the literal argument that was passed for <code>fun</code>. This works because of call-by-need semantics in R: function arguments are packaged up into <em>promises</em>. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">2</a></sup> discusses laziness in R in more detail.</p> + +<p>In this example, on line 8, we call <code>factory</code> with <code>function(a) x+a</code> as the argument for the formal parameter <code>fun</code>. Then, we evaluate that parse tree with <code>eval</code>.</p> + +<p>At this point, <code>res</code> is the closure with expression <code>function(a) x+a</code> and a reference to the environment of <code>makeTrivial</code>. On line 3, we change that reference to <code>baseenv()</code>, which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.</p> + +<p>Therefore, variable lookup in the function literal will only search the base environment, so <code>f()</code> results in an error.</p> + +<h3 id="static-scoping">Static scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeStatic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">globalenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeStatic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, on line 3, we update the environment of <code>res</code> to refer to <code>globalenv()</code>, which is the top-level environment where globals are defined.</p> + +<p>Therefore, variable lookup searches the top-level environment, so <code>f()</code> returns <code>0</code>.</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeLexical</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeLexical</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of <code>res</code> to <code>parent.frame()</code>, which is the environment of the function (<code>h</code>) that called the current function (<code>makeLexical</code>). This allows us to simulate lexical scoping, as if the function literal was evaluated inside <code>h</code>, rather than <code>makeLexical</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>h</code>, so <code>f()</code> returns <code>1</code>.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeDynamic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">function</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="nf">res</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> + <span class="p">}</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeDynamic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, we need another level of indirection. <code>makeDynamic</code> returns an anonymous function literal. The anonymous function takes <code>...</code>, which represents an arbitrary list of arguments, and then on line 5 we call <code>res</code> with those exact arguments. Note that we set the environment of <code>res</code> to be the environment of the <em>caller</em> of the anonymous function. Because of the multiple levels of indirection, the caller is <code>f</code>, on line 17.</p> + +<p>On line 12, <code>makeDynamic</code> returns a closure for the anonymous function. <code>h</code> returns that closure when it is called, and assigns it to <code>g</code>. When <code>g</code> is called on line 17, the function literal <code>function(a) x+a</code> is finally evaluated, and its environment is set to the environment of <code>f</code>, the caller of <code>g</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>f</code>, so <code>f()</code> returns <code>2</code>.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the <a href="/blog/2019/09/10/scoping-in-r/">previous post</a>, it&rsquo;s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.</p> + +<p>And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.</p> + +<p><em>I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>A. Goel and J. Vitek. &ldquo;On the Design, Implementation and Use of Laziness in R,&rdquo; in <em>Proceedings of the ACM in Programming Languages (PACMPL)</em>, vol. 3, no. OOPSLA, 2019. To appear. [<a href="http://janvitek.org/pubs/oopsla19a.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">↩</a></p></li></ol></div> + + Scoping in R + + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-scoping-in-r + 2019-09-10T10:00:00Z + 2019-09-10T10:00:00Z + + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">previous post</a> of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: <em>is R lexically or dynamically scoped?</em></p> +<!-- more--> + +<p>Recall the example program from before:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s examine what happens when we run this example. First, we create a mapping for <code>x</code> in the top-level environment. On line 2, we define a function <code>f</code>, which returns the value of some <code>x</code>. On line 3, we define a function <code>g</code>, which creates a new mapping for <code>x</code>, and then calls <code>f</code>. Note that the assignment on line 4 does <em>not</em> update the definition on line 1.</p> + +<p>When <code>f</code> is called, it needs to look up the value of <code>x</code>. In other words, does the reference of <code>x</code> on line 2 refer to the assignment on line 1 or the assignment on line 4? If <code>f</code> returns <code>1</code>, then the behaviour matches lexical scoping. If it returns <code>2</code>, then the behaviour matches dynamic scoping.</p> + +<p>When we run this example, the result is <code>1</code>. This implies that R is lexically scoped.</p> + +<p>But there&rsquo;s more to this story. In the rest of this blog post, I&rsquo;ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.</p> + +<p>The <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">next and final part</a> of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.</p> + +<h2 id="r-is-lexically-scoped-but">R is lexically scoped, but&hellip;</h2> + +<p>In <em>Evaluating the Design of the R Language</em>,<sup><a href="#2019-09-10-scoping-in-r-footnote-1-definition" name="2019-09-10-scoping-in-r-footnote-1-return">1</a></sup> Morandat, Hill, Osvald, and Vitek write:</p> + +<blockquote> + <p>As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.</p></blockquote> + +<p>In other words, R provides many different &ldquo;escape hatches&rdquo;&mdash;ways to bypass lexical scoping. Additionally, even without escape hatches, some of R&rsquo;s functionality can be surprising.</p> + +<h3 id="functions-environments-and-variables-in-r">Functions, environments, and variables in R</h3> + +<p>Before we look at some examples, I think it&rsquo;s useful to briefly discuss some of the core concepts in R that relate to scoping.</p> + +<ul> + <li> + <p><strong>Functions.</strong> R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.</p></li> + <li> + <p><strong>Environments.</strong> An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the &ldquo;enclosing&rdquo; environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.</p></li> + <li> + <p><strong>Variable lookup.</strong> When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.</p></li> + <li> + <p><strong>Variable assignment.</strong> <code>&lt;-</code> is the variable assignment operator in R. The expression <code>x &lt;- 1</code> assigns the value <code>1</code> to the variable <code>x</code> in the current environment. If a mapping for <code>x</code> already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.</p></li></ul> + +<p>From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.<sup><a href="#2019-09-10-scoping-in-r-footnote-2-definition" name="2019-09-10-scoping-in-r-footnote-2-return">2</a></sup></p> + +<p>On the other hand, variable lookup depends on the run-time state of the program&mdash;names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.</p> + +<p>The following examples will make this clear.</p> + +<h3 id="examples">Examples</h3> + +<h4 id="adding-variable-mappings-at-run-time">Adding variable mappings at run time</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it creates a function <code>g</code> that returns <code>x</code>, assigns <code>2</code> to <code>x</code>, and then calls <code>g</code>. When <code>g</code> is called, it looks up <code>x</code>. Since no mapping is found in <code>g</code>&rsquo;s environment, it searches in the enclosing environment (<code>f</code>&rsquo;s), and finds that <code>x</code> has value <code>2</code>. Therefore, <code>g</code> returns <code>2</code>.</p> + +<p>Note that the <code>x</code> on line 3 is resolved only when function <code>g</code> is called, not when it is defined. However, when <code>g</code> is defined, its environment has a reference to <code>f</code>&rsquo;s environment. Therefore, as long as <code>x</code> is defined <em>before</em> <code>g</code> is called, the lookup will always succeed.</p> + +<p>Here&rsquo;s a second example:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>f</code> is a function that branches on its argument, <code>b</code>. If <code>b</code> evaluates to true, then the expression <code>x &lt;- 2</code> is evaluated, and a mapping for <code>x</code> is created in <code>f</code>&rsquo;s environment. Otherwise, no mapping is created.</p> + +<p>When we look up the value of <code>x</code> on line 5, R will first search the function&rsquo;s environment. If <code>b</code> evaluated to true, then R will find a value for <code>x</code>, which is <code>2</code>. Otherwise, R will search in the enclosing environment of <code>f</code>, and find that <code>x</code> is <code>1</code>.</p> + +<p>Both of these examples vaguely resemble dynamic scoping, in that <code>x</code> takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.</p> + +<h4 id="function-lookup">Function lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> <span class="c1"># not an error</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">(</span><span class="m">42</span><span class="p">)</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.</p> + +<p>In this example, we call <code>g</code> with the argument <code>42</code>, which is not a function. Then, in the body of <code>g</code>, we call <code>f(0)</code> on line 3, which requires looking up <code>f</code>. Although there is an <code>f</code> in the environment of <code>g</code>, its value is <code>42</code>, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so <code>f(0)</code> returns <code>0</code>.</p> + +<p>This behaviour exists because <code>c</code> is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.</p> + +<h4 id="super-assignment">Super assignment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="n">x</span> <span class="o">&lt;&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +<span class="n">x</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>&lt;&lt;-</code> is the &ldquo;super assignment&rdquo; operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.</p> + +<p>In the above program, we define <code>x</code> to be <code>0</code> at the top level, and then define the function <code>f</code>. When we call <code>f</code> on line 7, it assigns <code>1</code> to <code>x</code> on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, <code>f</code> returns <code>x</code>, which is looked up from the local environment and has value <code>1</code>. Finally, line 8 looks up <code>x</code> from the top level environment, which has value <code>2</code>.</p> + +<h4 id="evaluating-arbitrary-code">Evaluating arbitrary code</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">eval</span><span class="p">(</span><span class="nf">parse</span><span class="p">(</span><span class="n">text</span> <span class="o">=</span> <span class="n">t</span><span class="p">))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="s">"x &lt;- 0"</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument <code>t</code>, which happens to be the string <code>"x &lt;- 0"</code>. Then, when line 4 executes, the lookup of <code>x</code> returns <code>0</code>.</p> + +<h4 id="simulating-dynamic-scope">Simulating dynamic scope</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.frame</span><span class="p">())</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 3, we perform an explicit variable lookup for <code>x</code>, but we do so in the environment <code>parent.frame()</code>, which refers to the calling function&rsquo;s environment, in this case, <code>g</code>&rsquo;s environment.. Therefore, the lookup returns <code>2</code>.</p> + +<p>Note that R has a similarly named function, <code>parent.env(e)</code> which returns the <em>enclosing environment</em> of the given environment <code>e</code>.</p> + +<h4 id="constructing-an-arbitrary-environment">Constructing an arbitrary environment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">e</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> + <span class="n">e</span><span class="o">$</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">3</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="n">e</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 3</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it constructs a new environment, <code>e</code>, which is initially empty. (By default, its enclosing environment is the current environment, which is <code>f</code>&rsquo;s.) Next, on line 4, it directly adds a mapping to that environment, assigning <code>3</code> to <code>x</code>. Then, on line 5, the lookup is explicitly done in environment <code>e</code>, so <code>f</code> returns <code>3</code>.</p> + +<h4 id="deleting-mappings">Deleting mappings</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">rm</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.env</span><span class="p">(</span><span class="nf">environment</span><span class="p">()))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># Error in f() : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Not only is it possible to dynamically add and modify mappings in R, but it is also possible to <em>delete</em> mappings. This is what line 3 does: it explicitly removes the mapping for <code>x</code> from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when <code>f</code> is called, the lookup of <code>x</code> fails and an error is raised.</p> + +<h4 id="infinite-loop-during-variable-lookup">Infinite loop during variable lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">enva</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="n">envb</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">enva</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">envb</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">envb</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="nf">environment</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="nf">f</span><span class="p">()</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.</p> + +<p>On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: <code>enva</code>&rsquo;s enclosing environment is <code>envb</code>, and <code>envb</code>&rsquo;s enclosing environment is <code>enva</code>.</p> + +<p>On line 5, we define a function with a free variable, <code>x</code>, but on line 6, we set <code>f</code>&rsquo;s environment to be <code>enva</code>. Finally, we call <code>f</code>.</p> + +<p>When the body of <code>f</code> is evaluated, it needs to look up <code>x</code>. Lookup starts in <code>f</code>&rsquo;s environment, which we set to be <code>enva</code>. Since no mapping for <code>x</code> is found, lookup continues in <code>enva</code>&rsquo;s enclosing environment, which is <code>envb</code>. However, <code>envb</code> is also empty, so lookup continues in its enclosing environment, which is <code>enva</code>, and now lookup results in an infinite loop.</p> + +<h3 id="an-intuition-for-scoping-in-r">An intuition for scoping in R</h3> + +<p>Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1"># example 1</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> + +<span class="c1"># example 2</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>It seems that <code>x</code> takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What&rsquo;s missing from our intuition?</p> + +<p>The key insight is that R is <em>function scoped</em>. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <code>&lt;-</code> <em>does not create a new scope</em>, and it is more useful to think of it as a mutation <em>on the current environment</em>. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)</p> + +<p>In a sense, it might be more accurate to say that R <em>environments</em> are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.</p> + +<h2 id="discussion">Discussion</h2> + +<p>All of this might make you a little uncomfortable, and uncertain about R&rsquo;s scoping rules.</p> + +<p>On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply &ldquo;lexical-like&rdquo; behaviour, and the creators have confirmed that lexical scoping was the intent.</p> + +<p>On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment&mdash;but this is not consistent with R&rsquo;s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>This ambiguity shows up in a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-3-definition" name="2019-09-10-scoping-in-r-footnote-3-return">3</a></sup> where the authors write:</p> + +<blockquote> + <p>Furthermore, because variable scoping in R is dynamic and can be modified at the language level [&hellip;] it cannot be trivially guaranteed that <code>x</code> is going to point to the same data structure throughout the entire execution of the loop.</p></blockquote> + +<p>It is true that a variable <code>x</code> may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.</p> + +<p>It is true that variable <em>lookup</em> is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of <em>dynamic scope</em>, then it would be fair to say that R is dynamically scoped.</p> + +<p>But if your definition of <em>dynamic scope</em> is &ldquo;a variable is bound to the most recent assignment during the program&rsquo;s execution,&rdquo; then it is not correct to say R is dynamically scoped.</p> + +<p>I think we have this ambiguity because <em>scope</em> (the places in a program where a variable can be referenced) and <em>variable lookup</em> or <em>name resolution</em> (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.</p> + +<p>Personally, I prefer the definition of <em>scope</em> that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is <em>both</em> lexically and dynamically scoped, or that R is <em>neither</em> lexically and dynamically scoped.</p> + +<p>I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language&mdash;when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.</p> + +<p>Regardless, it is very easy to miscommunicate, so I think it&rsquo;s important to be very clear and make sure you and your audience know what definitions of scoping you&rsquo;re using!</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>This entire adventure started when we were working on a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-4-definition" name="2019-09-10-scoping-in-r-footnote-4-return">4</a></sup> and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!</p> + +<p>This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.</p> + +<p>To summarize, I would say that <em>scope</em> refers to the places in a program where a variable is visible and can be referenced. Under <em>lexical scoping</em>, the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program. Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment during the program&rsquo;s execution.</p> + +<p>I would say that R <em>aims</em> to be lexically scoped&mdash;it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!</p> + +<p>If you want more content about R and scoping, the <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">third and final part</a> of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.</p> + +<p><strong>Edited 2020/02/21:</strong> For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), <a href="http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/">this blog post</a> has some nice examples and diagrams.</p> + +<p><em>I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.</em></p> + +<p><em>If you liked this post, you may also be interested in the following Twitter threads about R: <a href="https://twitter.com/mhyee/status/1063983175163158531">one</a>, <a href="https://twitter.com/mhyee/status/1067818720532316166">two</a> and <a href="https://twitter.com/mhyee/status/1074744049951739905">three</a>.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>F. Morandat, B. Hill, L. Osvald, J. Vitek. &ldquo;Evaluating the Design of the R Language,&rdquo; in <em>Proceedings of the European Conference on Object-Oriented Programming (ECOOP)</em>, 2012. [<a href="https://doi.org/10.1007/978-3-642-31057-7_6">DOI</a>][<a href="http://janvitek.org/pubs/ecoop12.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. &ldquo;Lexical Scope and Statistical Computing&rdquo;, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-2-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-3-definition" class="footnote-definition"> + <p>L. Stadler, A. Welc, C. Humer, and M. Jordan. &ldquo;Optimizing R Language Execution via Aggressive Speculation,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2016. [<a href="https://doi.org/10.1145/2989225.2989236">DOI</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-3-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-4-definition" class="footnote-definition"> + <p>O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. &ldquo;R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2019. To appear. [<a href="http://janvitek.org/pubs/dls19.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-4-return">↩</a></p></li></ol></div> + + Lexical and Dynamic Scope + + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + 2019-09-05T10:00:00Z + 2019-09-05T10:00:00Z + + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> \ No newline at end of file diff --git a/blog/feeds/scope.rss.xml b/blog/feeds/scope.rss.xml new file mode 100644 index 00000000..80c87366 --- /dev/null +++ b/blog/feeds/scope.rss.xml @@ -0,0 +1,1004 @@ + + + + PRL Blog: Posts tagged 'scope' + PRL Blog: Posts tagged 'scope' + http://prl.ccs.neu.edu/blog/tags/scope.html + Tue, 10 Sep 2019 11:00:00 UT + Tue, 10 Sep 2019 11:00:00 UT + 1800 + + Four Kinds of Scoping in R + http://prl.ccs.neu.edu/blog/2019/09/10/four-kinds-of-scoping-in-r/?utm_source=scope&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-four-kinds-of-scoping-in-r + Tue, 10 Sep 2019 11:00:00 UT + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">first</a> and <a href="/blog/2019/09/10/scoping-in-r/">second</a> parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.</p> + +<p>In this third and final part of my blog series, I&rsquo;d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.</p> + +<p>This is a &ldquo;bonus&rdquo; blog post, because I&rsquo;m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.</p> +<!-- more--> + +<h2 id="lexical-scope-and-statistical-computation">Lexical scope and statistical computation</h2> + +<p>In <em>Lexical Scope and Statistical Computation</em>,<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">1</a></sup> Robert Gentleman and Ross Ihaka, the creators of R, discuss why they designed R with lexical scoping. The paper is written for a statistics audience, and they provide motivating examples for having lexical scoping in R.</p> + +<p>For the purpose of their discussion, they define four (slightly different) kinds of scoping rules:</p> + +<ul> + <li><em>trivial</em>: no free variables allowed</li> + <li><em>static</em>: a free variable takes its value from a set of global variables</li> + <li><em>lexical</em>: a free variable takes the value of the binding that was in effect when the function was defined</li> + <li><em>dynamic</em>: a free variable takes the value of the most recent assignment to that variable</li></ul> + +<p>Note that under this set of definitions, <em>static scoping</em> is a separate scoping rule and not another name for <em>lexical scoping</em>.</p> + +<p>It is possible to simulate each of strategies in R. For fun, we can even construct &ldquo;factories&rdquo; that take a function, and then modify it to use the desired scoping rule! (Jan Ječmen originally provided these examples to me, and I adapted them for this blog post after some feedback from Artem Pelenitsyn.)</p> + +<h3 id="template">Template</h3> + +<p>Our examples will follow the template given below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">factory</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="o">&lt;???&gt;</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">factory</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># error, 0, 1, or 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We want to define a <code>factory</code> that takes a function literal and returns a closure that implements the desired scoping rule.</p> + +<p>Our example consists of three definitions of <code>x</code>. On line 5, we assign <code>0</code> to <code>x</code> at the top level. On line 7, we assign <code>1</code> to <code>x</code> inside function <code>h</code>, where we also create the closure. On line 12, we assign <code>2</code> to <code>x</code> inside the function <code>f</code> and right before we call <code>g</code>, which is the closure.</p> + +<p>Finally, we call <code>f</code> and observe the result:</p> + +<ul> + <li>Under trivial scoping, no free variables are allowed, so <code>f()</code> should result in an error.</li> + <li>Under static scoping, free variables may only refer to global variables, so <code>f()</code> should return <code>0</code>.</li> + <li>Under lexical scoping, free variables refer to the variables in scope when the function was defined, so <code>f()</code> should return <code>1</code>.</li> + <li>Under dynamic scoping, free variables take the value from the most recent assignment, so <code>f()</code> should return <code>2</code>.</li></ul> + +<p>We will implement the body of <code>factory</code> in only 3&ndash;5 lines of code. The rest of the code snippet, from lines 7 to the end, will remain the same, other than the call to <code>factory</code> on line 10.</p> + +<h3 id="trivial-scoping">Trivial scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeTrivial</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">baseenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeTrivial</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># Error in f(0) : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>substitute</code> returns the unevaluated parse tree for <code>fun</code>. In other words, it obtains the literal argument that was passed for <code>fun</code>. This works because of call-by-need semantics in R: function arguments are packaged up into <em>promises</em>. As a result, the syntax tree of arguments is available for metaprogramming. A recent paper by Goel and Vitek<sup><a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" name="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">2</a></sup> discusses laziness in R in more detail.</p> + +<p>In this example, on line 8, we call <code>factory</code> with <code>function(a) x+a</code> as the argument for the formal parameter <code>fun</code>. Then, we evaluate that parse tree with <code>eval</code>.</p> + +<p>At this point, <code>res</code> is the closure with expression <code>function(a) x+a</code> and a reference to the environment of <code>makeTrivial</code>. On line 3, we change that reference to <code>baseenv()</code>, which is the environment containing library definitions. Since this environment is above the (user) top-level environment, global variables are not available.</p> + +<p>Therefore, variable lookup in the function literal will only search the base environment, so <code>f()</code> results in an error.</p> + +<h3 id="static-scoping">Static scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeStatic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">globalenv</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeStatic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, on line 3, we update the environment of <code>res</code> to refer to <code>globalenv()</code>, which is the top-level environment where globals are defined.</p> + +<p>Therefore, variable lookup searches the top-level environment, so <code>f()</code> returns <code>0</code>.</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeLexical</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="n">res</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeLexical</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Although lexical scoping is the default for R, our factory template requires some metaprogramming to work properly. We need to set the environment of <code>res</code> to <code>parent.frame()</code>, which is the environment of the function (<code>h</code>) that called the current function (<code>makeLexical</code>). This allows us to simulate lexical scoping, as if the function literal was evaluated inside <code>h</code>, rather than <code>makeLexical</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>h</code>, so <code>f()</code> returns <code>1</code>.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">makeDynamic</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">fun</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">function</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> <span class="p">{</span> + <span class="n">res</span> <span class="o">&lt;-</span> <span class="nf">eval</span><span class="p">(</span><span class="nf">substitute</span><span class="p">(</span><span class="n">fun</span><span class="p">))</span> + <span class="nf">environment</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="nf">parent.frame</span><span class="p">()</span> + <span class="nf">res</span><span class="p">(</span><span class="kc">...</span><span class="p">)</span> + <span class="p">}</span> +<span class="p">}</span> + +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">h</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="nf">makeDynamic</span><span class="p">(</span><span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span><span class="o">+</span><span class="n">a</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">h</span><span class="p">()</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>For this example, we need another level of indirection. <code>makeDynamic</code> returns an anonymous function literal. The anonymous function takes <code>...</code>, which represents an arbitrary list of arguments, and then on line 5 we call <code>res</code> with those exact arguments. Note that we set the environment of <code>res</code> to be the environment of the <em>caller</em> of the anonymous function. Because of the multiple levels of indirection, the caller is <code>f</code>, on line 17.</p> + +<p>On line 12, <code>makeDynamic</code> returns a closure for the anonymous function. <code>h</code> returns that closure when it is called, and assigns it to <code>g</code>. When <code>g</code> is called on line 17, the function literal <code>function(a) x+a</code> is finally evaluated, and its environment is set to the environment of <code>f</code>, the caller of <code>g</code>.</p> + +<p>Therefore, variable lookup searches the environment of <code>f</code>, so <code>f()</code> returns <code>2</code>.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Hopefully this blog post has shown another way of looking at scoping definitions. As discussed in the <a href="/blog/2019/09/10/scoping-in-r/">previous post</a>, it&rsquo;s very easy to get confused because different definitions are used by different people. Here, Gentleman and Ihaka very clearly state what definitions they are using.</p> + +<p>And finally, while I am far from an expert on metaprogramming in R, I hope this post has given a taste of what is possible.</p> + +<p><em>I would like to thank Jan Ječmen for coming up with and showing me the original versions of these code examples, and Artem Pelenitsyn for his feedback to improve and not discard these examples from an earlier blog draft.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. "Lexical Scope and Statistical Computing, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-four-kinds-of-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>A. Goel and J. Vitek. &ldquo;On the Design, Implementation and Use of Laziness in R,&rdquo; in <em>Proceedings of the ACM in Programming Languages (PACMPL)</em>, vol. 3, no. OOPSLA, 2019. To appear. [<a href="http://janvitek.org/pubs/oopsla19a.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-four-kinds-of-scoping-in-r-footnote-2-return">↩</a></p></li></ol></div> + + Scoping in R + http://prl.ccs.neu.edu/blog/2019/09/10/scoping-in-r/?utm_source=scope&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-10-scoping-in-r + Tue, 10 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>In the <a href="/blog/2019/09/05/lexical-and-dynamic-scope/">previous post</a> of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: <em>is R lexically or dynamically scoped?</em></p> +<!-- more--> + +<p>Recall the example program from before:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s examine what happens when we run this example. First, we create a mapping for <code>x</code> in the top-level environment. On line 2, we define a function <code>f</code>, which returns the value of some <code>x</code>. On line 3, we define a function <code>g</code>, which creates a new mapping for <code>x</code>, and then calls <code>f</code>. Note that the assignment on line 4 does <em>not</em> update the definition on line 1.</p> + +<p>When <code>f</code> is called, it needs to look up the value of <code>x</code>. In other words, does the reference of <code>x</code> on line 2 refer to the assignment on line 1 or the assignment on line 4? If <code>f</code> returns <code>1</code>, then the behaviour matches lexical scoping. If it returns <code>2</code>, then the behaviour matches dynamic scoping.</p> + +<p>When we run this example, the result is <code>1</code>. This implies that R is lexically scoped.</p> + +<p>But there&rsquo;s more to this story. In the rest of this blog post, I&rsquo;ll examine some interesting scoping examples in R, and discuss how the scoping definitions relate to R.</p> + +<p>The <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">next and final part</a> of this blog series, published simultaneously with this one, is an appendix where I implement four different scoping disciplines in R.</p> + +<h2 id="r-is-lexically-scoped-but">R is lexically scoped, but&hellip;</h2> + +<p>In <em>Evaluating the Design of the R Language</em>,<sup><a href="#2019-09-10-scoping-in-r-footnote-1-definition" name="2019-09-10-scoping-in-r-footnote-1-return">1</a></sup> Morandat, Hill, Osvald, and Vitek write:</p> + +<blockquote> + <p>As is often the case, R is lexically scoped up to the point it is not. R is above all a dynamic language with full reflective access to the running program’s data and representation.</p></blockquote> + +<p>In other words, R provides many different &ldquo;escape hatches&rdquo;&mdash;ways to bypass lexical scoping. Additionally, even without escape hatches, some of R&rsquo;s functionality can be surprising.</p> + +<h3 id="functions-environments-and-variables-in-r">Functions, environments, and variables in R</h3> + +<p>Before we look at some examples, I think it&rsquo;s useful to briefly discuss some of the core concepts in R that relate to scoping.</p> + +<ul> + <li> + <p><strong>Functions.</strong> R has first-class functions, and functions evaluate to closures. In other words, a function value includes both the body of the function as well as the environment that the function was evaluated in. In R, the programmer can modify the environment of a closure. Note that R is function scoped; there is no block scoping.</p></li> + <li> + <p><strong>Environments.</strong> An environment in R is a mapping from variables to values. Each function has its own local environment. Furthermore, each environment has a reference to the &ldquo;enclosing&rdquo; environment that it was evaluated in. R environments are first-class, meaning the programmer can add, modify, or removing variable mappings, and also change the reference to the enclosing environment.</p></li> + <li> + <p><strong>Variable lookup.</strong> When R looks up a variable, it will search in the current environment for a mapping. If no mapping is found, then it will search in the enclosing environment. This process continues until a mapping is found, or the topmost, empty environment is reached, in which case an error is raised.</p></li> + <li> + <p><strong>Variable assignment.</strong> <code>&lt;-</code> is the variable assignment operator in R. The expression <code>x &lt;- 1</code> assigns the value <code>1</code> to the variable <code>x</code> in the current environment. If a mapping for <code>x</code> already exists in the environment, then the assignment will update and overwrite the existing value. Otherwise, a new mapping is created in the environment. Note that variable assignment can only update the current environment, and never creates a scope.</p></li></ul> + +<p>From this description, we can see that R implements lexical scoping (or at least, something that behaves a lot like lexical scoping): each function value is associated with the environment it was evaluated in, and variable lookup proceeds along the chain of enclosing environments. In fact, the creators of R have confirmed that lexical scoping was their intent.<sup><a href="#2019-09-10-scoping-in-r-footnote-2-definition" name="2019-09-10-scoping-in-r-footnote-2-return">2</a></sup></p> + +<p>On the other hand, variable lookup depends on the run-time state of the program&mdash;names cannot be resolved statically. Furthermore, since R provides operations for environment manipulation, a programmer can easily circumvent lexical scoping.</p> + +<p>The following examples will make this clear.</p> + +<h3 id="examples">Examples</h3> + +<h4 id="adding-variable-mappings-at-run-time">Adding variable mappings at run time</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it creates a function <code>g</code> that returns <code>x</code>, assigns <code>2</code> to <code>x</code>, and then calls <code>g</code>. When <code>g</code> is called, it looks up <code>x</code>. Since no mapping is found in <code>g</code>&rsquo;s environment, it searches in the enclosing environment (<code>f</code>&rsquo;s), and finds that <code>x</code> has value <code>2</code>. Therefore, <code>g</code> returns <code>2</code>.</p> + +<p>Note that the <code>x</code> on line 3 is resolved only when function <code>g</code> is called, not when it is defined. However, when <code>g</code> is defined, its environment has a reference to <code>f</code>&rsquo;s environment. Therefore, as long as <code>x</code> is defined <em>before</em> <code>g</code> is called, the lookup will always succeed.</p> + +<p>Here&rsquo;s a second example:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>f</code> is a function that branches on its argument, <code>b</code>. If <code>b</code> evaluates to true, then the expression <code>x &lt;- 2</code> is evaluated, and a mapping for <code>x</code> is created in <code>f</code>&rsquo;s environment. Otherwise, no mapping is created.</p> + +<p>When we look up the value of <code>x</code> on line 5, R will first search the function&rsquo;s environment. If <code>b</code> evaluated to true, then R will find a value for <code>x</code>, which is <code>2</code>. Otherwise, R will search in the enclosing environment of <code>f</code>, and find that <code>x</code> is <code>1</code>.</p> + +<p>Both of these examples vaguely resemble dynamic scoping, in that <code>x</code> takes the value of the most recent assignment. However, this is not how R is implemented, and it is not consistent with how R behaves in other examples.</p> + +<h4 id="function-lookup">Function lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> <span class="c1"># not an error</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">(</span><span class="m">42</span><span class="p">)</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has slightly different lookup rules, if the variable is in function call position. Specifically, R will search the environment chain and skip non-function values.</p> + +<p>In this example, we call <code>g</code> with the argument <code>42</code>, which is not a function. Then, in the body of <code>g</code>, we call <code>f(0)</code> on line 3, which requires looking up <code>f</code>. Although there is an <code>f</code> in the environment of <code>g</code>, its value is <code>42</code>, which is not a function. R will then search the enclosing environment, where it finds the function defined on line 1. Therefore, the lookup on line 3 resolves to the function on line 1, so <code>f(0)</code> returns <code>0</code>.</p> + +<p>This behaviour exists because <code>c</code> is the built-in function that constructs vectors (in other words, one of the most commonly used functions in R), but it is also a commonly used argument name.</p> + +<h4 id="super-assignment">Super assignment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">0</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> + <span class="n">x</span> <span class="o">&lt;&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 1</span> +<span class="n">x</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p><code>&lt;&lt;-</code> is the &ldquo;super assignment&rdquo; operator. It skips the current environment and then searches the chain of enclosing environments until it finds a variable to update. If no variable is found, then a new mapping is created at the top environment.</p> + +<p>In the above program, we define <code>x</code> to be <code>0</code> at the top level, and then define the function <code>f</code>. When we call <code>f</code> on line 7, it assigns <code>1</code> to <code>x</code> on line 3, which creates a mapping in the local environment. On line 4, the super assignment skips the mapping in the local environment and instead updates the mapping created on line 1. Next, <code>f</code> returns <code>x</code>, which is looked up from the local environment and has value <code>1</code>. Finally, line 8 looks up <code>x</code> from the top level environment, which has value <code>2</code>.</p> + +<h4 id="evaluating-arbitrary-code">Evaluating arbitrary code</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">eval</span><span class="p">(</span><span class="nf">parse</span><span class="p">(</span><span class="n">text</span> <span class="o">=</span> <span class="n">t</span><span class="p">))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="s">"x &lt;- 0"</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 0</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>R has a mechanism for converting an arbitrary string to code and then executing it. On line 3, we parse and evaluate the argument <code>t</code>, which happens to be the string <code>"x &lt;- 0"</code>. Then, when line 4 executes, the lookup of <code>x</code> returns <code>0</code>.</p> + +<h4 id="simulating-dynamic-scope">Simulating dynamic scope</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.frame</span><span class="p">())</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 2</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 3, we perform an explicit variable lookup for <code>x</code>, but we do so in the environment <code>parent.frame()</code>, which refers to the calling function&rsquo;s environment, in this case, <code>g</code>&rsquo;s environment.. Therefore, the lookup returns <code>2</code>.</p> + +<p>Note that R has a similarly named function, <code>parent.env(e)</code> which returns the <em>enclosing environment</em> of the given environment <code>e</code>.</p> + +<h4 id="constructing-an-arbitrary-environment">Constructing an arbitrary environment</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">e</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> + <span class="n">e</span><span class="o">$</span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">3</span> + <span class="nf">get</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="n">e</span><span class="p">)</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># 3</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>When <code>f</code> is called, it constructs a new environment, <code>e</code>, which is initially empty. (By default, its enclosing environment is the current environment, which is <code>f</code>&rsquo;s.) Next, on line 4, it directly adds a mapping to that environment, assigning <code>3</code> to <code>x</code>. Then, on line 5, the lookup is explicitly done in environment <code>e</code>, so <code>f</code> returns <code>3</code>.</p> + +<h4 id="deleting-mappings">Deleting mappings</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="nf">rm</span><span class="p">(</span><span class="s">"x"</span><span class="p">,</span> <span class="n">envir</span> <span class="o">=</span> <span class="nf">parent.env</span><span class="p">(</span><span class="nf">environment</span><span class="p">()))</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># Error in f() : object &#39;x&#39; not found</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Not only is it possible to dynamically add and modify mappings in R, but it is also possible to <em>delete</em> mappings. This is what line 3 does: it explicitly removes the mapping for <code>x</code> from the enclosing environment of the current environment. In other words, the definition on line 1 is deleted. Therefore, when <code>f</code> is called, the lookup of <code>x</code> fails and an error is raised.</p> + +<h4 id="infinite-loop-during-variable-lookup">Infinite loop during variable lookup</h4> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">enva</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="n">envb</span> <span class="o">&lt;-</span> <span class="nf">new.env</span><span class="p">()</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">enva</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">envb</span> +<span class="nf">parent.env</span><span class="p">(</span><span class="n">envb</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> +<span class="nf">environment</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">&lt;-</span> <span class="n">enva</span> +<span class="nf">f</span><span class="p">()</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In this final example, manipulation of environments allows us to create a function where variable lookup results in an infinite loop.</p> + +<p>On lines 1 and 2, we create new, empty environments. Both have the same enclosing environment, which is the top-level environment. However, on lines 3 and 4, we modify their enclosing environments to create a cycle: <code>enva</code>&rsquo;s enclosing environment is <code>envb</code>, and <code>envb</code>&rsquo;s enclosing environment is <code>enva</code>.</p> + +<p>On line 5, we define a function with a free variable, <code>x</code>, but on line 6, we set <code>f</code>&rsquo;s environment to be <code>enva</code>. Finally, we call <code>f</code>.</p> + +<p>When the body of <code>f</code> is evaluated, it needs to look up <code>x</code>. Lookup starts in <code>f</code>&rsquo;s environment, which we set to be <code>enva</code>. Since no mapping for <code>x</code> is found, lookup continues in <code>enva</code>&rsquo;s enclosing environment, which is <code>envb</code>. However, <code>envb</code> is also empty, so lookup continues in its enclosing environment, which is <code>enva</code>, and now lookup results in an infinite loop.</p> + +<h3 id="an-intuition-for-scoping-in-r">An intuition for scoping in R</h3> + +<p>Some of the above examples appear to demonstrate dynamic scoping. Recall two of our examples:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1"># example 1</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="n">x</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">g</span><span class="p">()</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">()</span> <span class="c1"># 2</span> + +<span class="c1"># example 2</span> +<span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">{</span> + <span class="nf">if </span><span class="p">(</span><span class="n">b</span><span class="p">)</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="n">x</span> +<span class="p">}</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">TRUE</span><span class="p">)</span> <span class="c1"># 2</span> +<span class="nf">f</span><span class="p">(</span><span class="kc">FALSE</span><span class="p">)</span> <span class="c1"># 1</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>It seems that <code>x</code> takes on the value of the last assignment, but we know this is not the case, from the first example. This is also not how R is implemented. What&rsquo;s missing from our intuition?</p> + +<p>The key insight is that R is <em>function scoped</em>. In R, each function has an associated environment, and that environment implements a scope. In general, only a function definition can create a scope. Therefore, the assignment operator <code>&lt;-</code> <em>does not create a new scope</em>, and it is more useful to think of it as a mutation <em>on the current environment</em>. (In contrast, in most languages, a variable binding or definition creates a new scope, and an assignment mutates that variable.)</p> + +<p>In a sense, it might be more accurate to say that R <em>environments</em> are lexically scoped, variables are scoped to functions (but a reference can occur syntactically before a definition), and variable assignment is an update to the environment.</p> + +<h2 id="discussion">Discussion</h2> + +<p>All of this might make you a little uncomfortable, and uncertain about R&rsquo;s scoping rules.</p> + +<p>On one hand, R passes the first example program as a lexically scoped language, the implementation of closures and variable lookup imply &ldquo;lexical-like&rdquo; behaviour, and the creators have confirmed that lexical scoping was the intent.</p> + +<p>On the other hand, variable lookup depends on the run-time state of the program, and variable bindings cannot be resolved statically. Some of the examples even resemble dynamic scoping, where a free variable takes the value of the most recent assignment&mdash;but this is not consistent with R&rsquo;s behaviour in other examples. Furthermore, the dynamic nature of R and its reflection and metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>This ambiguity shows up in a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-3-definition" name="2019-09-10-scoping-in-r-footnote-3-return">3</a></sup> where the authors write:</p> + +<blockquote> + <p>Furthermore, because variable scoping in R is dynamic and can be modified at the language level [&hellip;] it cannot be trivially guaranteed that <code>x</code> is going to point to the same data structure throughout the entire execution of the loop.</p></blockquote> + +<p>It is true that a variable <code>x</code> may not point to the same data structure during the execution of a loop. It is true that scoping in R can be modified at the language level.</p> + +<p>It is true that variable <em>lookup</em> is dynamic, as it is performed at run time and depends on the run-time program state. If that is your definition of <em>dynamic scope</em>, then it would be fair to say that R is dynamically scoped.</p> + +<p>But if your definition of <em>dynamic scope</em> is &ldquo;a variable is bound to the most recent assignment during the program&rsquo;s execution,&rdquo; then it is not correct to say R is dynamically scoped.</p> + +<p>I think we have this ambiguity because <em>scope</em> (the places in a program where a variable can be referenced) and <em>variable lookup</em> or <em>name resolution</em> (determining which binding or definition a name refers to) are often considered together. For most lexically scoped languages, name resolution can be done at compile time. For most dynamically scoped languages, name resolution must be done at run time. R is lexically scoped, but must perform name resolution at run time.</p> + +<p>Personally, I prefer the definition of <em>scope</em> that treats name resolution as an orthogonal issue. I think it is more useful to keep the two issues separate. In addition, I think it is confusing and unhelpful to say that R is <em>both</em> lexically and dynamically scoped, or that R is <em>neither</em> lexically and dynamically scoped.</p> + +<p>I think it is more helpful to treat R as a lexically scoped language (with certain exceptions and surprises) than as a dynamically scoped language&mdash;when I read and write R code, I find it more convenient to think about nested function definitions and free variables in terms of lexical scoping rules. And I think that it is more accurate, based on the design and implementation, to classify R as a lexically scoped language.</p> + +<p>Regardless, it is very easy to miscommunicate, so I think it&rsquo;s important to be very clear and make sure you and your audience know what definitions of scoping you&rsquo;re using!</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>This entire adventure started when we were working on a paper,<sup><a href="#2019-09-10-scoping-in-r-footnote-4-definition" name="2019-09-10-scoping-in-r-footnote-4-return">4</a></sup> and asked each other, is R lexically or dynamically scoped? Eventually, it became apparent that we had different definitions of lexical and dynamic scope, so of course we were unable to agree on an answer!</p> + +<p>This got me interested in exploring definitions of scope, the history of lexical scope, and how R fits with traditional definitions of lexical scope. The result was this mini blog series.</p> + +<p>To summarize, I would say that <em>scope</em> refers to the places in a program where a variable is visible and can be referenced. Under <em>lexical scoping</em>, the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program. Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment during the program&rsquo;s execution.</p> + +<p>I would say that R <em>aims</em> to be lexically scoped&mdash;it was part of the design and implementation, but certain features make the situation more complicated. In particular, variables are function scoped, definitions do not introduce new scopes, and variable lookup is performed at run time. Furthermore, the dynamic nature of R and its metaprogramming capabilities allow programmers to completely circumvent lexical scoping.</p> + +<p>Finally, there are some definitions of lexical and dynamic scope that also consider variable lookup. Under these definitions, R might be considered a dynamically scoped language, since variable lookup happens at run time. Therefore, it is important to be precise about your definitions!</p> + +<p>If you want more content about R and scoping, the <a href="/blog/2019/09/10/four-kinds-of-scoping-in-r/">third and final part</a> of this blog series is already published. In it, I walk through four different examples of using metaprogramming to simulate different scoping disciplines in R.</p> + +<p><strong>Edited 2020/02/21:</strong> For another discussion on R environments and lookups, (and also packages and namespaces, which I did not cover in my post), <a href="http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/">this blog post</a> has some nice examples and diagrams.</p> + +<p><em>I would like to thank Sam Caldwell, Guido Chari, Oli Flückiger, Aviral Goel, Ben Greenman, Jakob Hain, Jan Ječmen, Hugo Musso Gualandi, Artem Pelenitsyn, and Jan Vitek for their comments, feedback, and discussions that have greatly improved and shaped this blog post.</em></p> + +<p><em>If you liked this post, you may also be interested in the following Twitter threads about R: <a href="https://twitter.com/mhyee/status/1063983175163158531">one</a>, <a href="https://twitter.com/mhyee/status/1067818720532316166">two</a> and <a href="https://twitter.com/mhyee/status/1074744049951739905">three</a>.</em></p> + +<hr /> + +<h2 id="references">References</h2> + +<div class="footnotes"> + <ol> + <li id="2019-09-10-scoping-in-r-footnote-1-definition" class="footnote-definition"> + <p>F. Morandat, B. Hill, L. Osvald, J. Vitek. &ldquo;Evaluating the Design of the R Language,&rdquo; in <em>Proceedings of the European Conference on Object-Oriented Programming (ECOOP)</em>, 2012. [<a href="https://doi.org/10.1007/978-3-642-31057-7_6">DOI</a>][<a href="http://janvitek.org/pubs/ecoop12.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-1-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-2-definition" class="footnote-definition"> + <p>R. Gentleman and R. Ihaka. &ldquo;Lexical Scope and Statistical Computing&rdquo;, <em>Journal of Computational and Graphical Statistics</em>, vol. 9, no. 3, 2000. [<a href="https://doi.org/10.1080/10618600.2000.10474895">DOI</a>][<a href="https://www.stat.auckland.ac.nz/~ihaka/downloads/lexical.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-2-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-3-definition" class="footnote-definition"> + <p>L. Stadler, A. Welc, C. Humer, and M. Jordan. &ldquo;Optimizing R Language Execution via Aggressive Speculation,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2016. [<a href="https://doi.org/10.1145/2989225.2989236">DOI</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-3-return">↩</a></p></li> + <li id="2019-09-10-scoping-in-r-footnote-4-definition" class="footnote-definition"> + <p>O. Flückiger, G. Chari, J. Ječmen, M.-H. Yee, J. Hain, and J. Vitek. &ldquo;R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments,&rdquo; in <em>Proceedings of the Symposium on Dynamic Languages (DLS)</em>, 2019. To appear. [<a href="http://janvitek.org/pubs/dls19.pdf">Available online</a>]&nbsp;<a href="#2019-09-10-scoping-in-r-footnote-4-return">↩</a></p></li></ol></div> + + Lexical and Dynamic Scope + http://prl.ccs.neu.edu/blog/2019/09/05/lexical-and-dynamic-scope/?utm_source=scope&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-09-05-lexical-and-dynamic-scope + Thu, 05 Sep 2019 10:00:00 UT + Ming-Ho Yee + +<p>This all started with a simple question about the R programming language: <em>is R lexically or dynamically scoped?</em></p> + +<p>To answer that question, we need to understand what <em>scope</em> is, along with <em>lexical scope</em> and <em>dynamic scope</em>.</p> +<!-- more--> + +<p>In this blog post, I&rsquo;d like to explain the differences between lexical scope and dynamic scope, and also explore some of the history behind those ideas. In a <a href="/blog/2019/09/10/scoping-in-r/">subsequent post</a>, I&rsquo;ll discuss scoping in R and why it can be confusing.</p> + +<h2 id="what-is-scope">What is scope?</h2> + +<p><em>Scope</em> refers to the places in a program where a variable is visible and can be referenced.</p> + +<p>An interesting situation is when a function has free variables. Consider the example below:</p> + +<div class="brush: r"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">x</span> <span class="o">&lt;-</span> <span class="m">1</span> +<span class="n">f</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">x</span> <span class="o">+</span> <span class="n">a</span> +<span class="n">g</span> <span class="o">&lt;-</span> <span class="nf">function</span><span class="p">()</span> <span class="p">{</span> + <span class="n">x</span> <span class="o">&lt;-</span> <span class="m">2</span> + <span class="nf">f</span><span class="p">(</span><span class="m">0</span><span class="p">)</span> +<span class="p">}</span> +<span class="nf">g</span><span class="p">()</span> <span class="c1"># what does this return?</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>On line 1, we create a mapping for <code>x</code> with value <code>1</code>. On line 2, we define a function <code>f</code> whose body uses the parameter <code>a</code>, but also the free variable <code>x</code>. On line 3, we define a function <code>g</code>, whose body creates a new mapping for <code>x</code> with value <code>2</code>, and then calls <code>f(0)</code>. (Note that line 4 does not update the mapping created on line 1.) Finally, on line 7, we call <code>g()</code>.</p> + +<p>What value does <code>g</code> return when it is called? What mapping does the free variable <code>x</code> on line 2 refer to? Does it refer to the mapping on line 1 that was visible when <code>f</code> was defined? Or does it refer to the mapping on line 4 that was created just before <code>f</code> was called?</p> + +<h3 id="lexical-scoping">Lexical scoping</h3> + +<p>Under <em>lexical scoping</em> (also known as <em>static scoping</em>), the scope of a variable is determined by the lexical (<em>i.e.</em>, textual) structure of a program.</p> + +<p>In the example above, the definition of <code>x</code> on line 1 creates a scope that starts after its definition and extends <em>into</em> the bodies of <code>f</code> and <code>g</code>. However, the second definition of <code>x</code> on line 4 creates a new scope that (1) shadows the previous definition of <code>x</code>, and (2) does not extend into the call <code>f(0)</code> on line 5. Looking at this from another direction, the use of <code>x</code> on line 2 is within the scope created by the definition on line 1, and thus refers to that definition.</p> + +<p>Therefore, under lexical scoping, the example program returns <code>1</code>.</p> + +<p>Most programming languages we use today are lexically scoped. Intuitively, a human (or compiler) can determine the scope of a variable by just examining the source code of a program. In other words, a compiler can determine which <em>definition</em> each variable refers to&mdash;but it may not be able to determine the <em>values</em> of each variable.</p> + +<h3 id="dynamic-scoping">Dynamic scoping</h3> + +<p>Under <em>dynamic scoping</em>, a variable is bound to the most recent value assigned to that variable, <em>i.e.</em>, the most recent assignment <em>during the program&rsquo;s execution</em>.</p> + +<p>In the example above, the free variable <code>x</code> in the body of <code>f</code> is evaluated when <code>f(0)</code> is called on line 5. At that point (during program execution), the most recent assignment was on line 4.</p> + +<p>Therefore, under dynamic scoping, the example program returns <code>2</code>.</p> + +<p>Dynamically scoped programming languages include bash, LaTeX, and the original version of Lisp. Emacs Lisp is dynamically scoped, but allows the programmer to select lexical scoping. Conversely, Perl and Common Lisp are lexically scoped by default, but allow the programmer to select dynamic scoping.</p> + +<p>(<strong>Edited 2020/08/13:</strong> As of <a href="https://www.gnu.org/savannah-checkouts/gnu/emacs/news/NEWS.27.1">Emacs 27.1</a>, &ldquo;lexical binding is now used by default when evaluating interactive Elisp.&rdquo; Thanks to Artem Pelenitsyn for bringing this to my attention.)</p> + +<h2 id="now-for-a-digression">Now for a digression</h2> + +<p>These are the definitions I learned from my classes and textbooks, and should be similar to other definitions and explanations you might find online.</p> + +<p>However, it took me many drafts and attempts before arriving at the current version. I had difficulty writing an explanation that I was satisfied with&mdash;a definition that was not circular, did not appeal to some intuition or familiarity, and did not conflate terms. Even some of the resources I consulted had these issues.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-1-return">1</a></sup></p> + +<p>I am much happier with my current version, but it still bothers me slightly. If lexical scope and dynamic scope are related concepts, then why are the definitions so different? Why does the definition for <em>dynamic scope</em> not mention scope at all? If <em>scope</em> is about &ldquo;where a variable is visible,&rdquo; and that definition is with respect to a <em>variable definition</em>, then why do so many explanations and examples define lexical and dynamic scope in terms of <em>variable use</em>?</p> + +<h2 id="scope-and-extent">Scope and Extent</h2> + +<p>I found some answers in Guy Steele&rsquo;s <em>Common Lisp the Language, 2nd Edition</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-2-return">2</a></sup> which Matthias Felleisen recommended to me.</p> + +<p>In chapter 3, Steele introduces the concepts of <em>scope</em> and <em>extent</em>:</p> + +<blockquote> + <p><em>Scope</em> refers to the spatial or textual region of the program within which references may occur. <em>Extent</em> refers to the interval of time during which references may occur.</p></blockquote> + +<p>In addition, there are four interesting cases of scope and extent, with respect to Common Lisp:</p> + +<ul> + <li> + <p><em>Lexical scope</em>: a reference can only occur within certain textual regions of the program, which are determined by the establishing construct, <em>e.g.</em>, the body of a variable definition.</p></li> + <li> + <p><em>Indefinite scope</em>: a reference can occur anywhere in the program.</p></li> + <li> + <p><em>Dynamic extent</em>: a reference can occur during the time between an entity&rsquo;s creation and its explicit destruction, <em>e.g.</em>, when a local variable is created upon entering a function and destroyed when returning from that function.</p></li> + <li> + <p><em>Indefinite extent</em>: an entity may exist as long as it is possible to be referenced. (Note that this is the idea behind garbage collection: an entity can be destroyed once references to it are impossible.)</p></li></ul> + +<p>Steele points out that <em>dynamic scope</em> is a misnomer, even though it is both a traditional and useful concept. It can be defined as <em>indefinite scope and dynamic extent</em>. In other words, references to a variable may occur anywhere in a program, as long as that variable has been initialized and has not yet been explicitly destroyed. Furthermore, a later initialization hides an earlier one.</p> + +<h3 id="discussion">Discussion</h3> + +<p>I found this approach very informative, because it explicitly distinguishes between space (scope) and time (extent), which further implies a separation between compile time and run time. This explains my unease with the definition of &ldquo;dynamic scope&rdquo;&mdash;it is nominally about textual regions in a program, but also requires consideration of run-time behaviour. Dynamic scope is a misnomer!</p> + +<p>The above definitions are specifically for Common Lisp, but I believe we can learn from them and adapt them for other programming languages.</p> + +<h2 id="a-brief-and-incomplete-history-of-lexical-scope">A brief and incomplete history of lexical scope</h2> + +<p>During my research of different definitions of lexical scope, I began to wonder if there was an &ldquo;original&rdquo; definition of lexical scope. I did not find one, but I was able to trace some of the connections between Lisp, Scheme, and ALGOL 60. This history is certainly incomplete, but I hope it is somewhat useful and interesting.</p> + +<ul> + <li> + <p><strong>1960</strong>. John McCarthy publishes the original paper on Lisp.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-3-return">3</a></sup> In <em>History of Lisp</em>,<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-4-return">4</a></sup> McCarthy writes that he borrowed the λ-notation from Alonzo Church&rsquo;s lambda calculus, but none of the other ideas. He also recounts an incident where a programmer desired lexical scoping, but Lisp used dynamic scoping. McCarthy considered this to be a bug, which Steve Russell later fixed by developing the &ldquo;FUNARG device.&rdquo;</p></li> + <li> + <p><strong>1963</strong>. After a few years of work, the <em>Revised Report on Algorithm Language ALGOL 60</em> is published.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-5-return">5</a></sup> While &ldquo;lexical scope&rdquo; is not explicitly mentioned, it is recognizable in the specification.</p></li> + <li> + <p><strong>1964</strong>. Peter Landin shows how expressions in programming languages can be modelled in Church&rsquo;s λ-notation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-6-return">6</a></sup> He also introduces the concept of a <em>closure</em>, which pairs a lambda expression with the environment it was evaluated in.</p></li> + <li> + <p><strong>1970</strong>. Joel Moses describes the problem of free variables in functions.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-7-return">7</a></sup> He considers both the &ldquo;downward&rdquo; case (where a function is passed to another function) and the &ldquo;upward&rdquo; case (where a function returns a function), and remarks on the correspondence between Lisp&rsquo;s FUNARG device and Landin&rsquo;s closures.</p></li> + <li> + <p><strong>1975</strong>. Gerald Sussman and Guy Steele publish the first Scheme paper.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-8-return">8</a></sup> They describe their goal of a Lisp-like language that is based on the lambda calculus. As a consequence, they implement lexical scoping with closures, to preserve the substitution semantics of the lambda calculus. They compare this scoping discipline to ALGOL&rsquo;s.</p></li> + <li> + <p><strong>1978</strong>. Steele and Sussman describe various programming language design choices, by developing an interpreter for each programming language variation.<sup><a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" name="2019-09-05-lexical-and-dynamic-scope-footnote-9-return">9</a></sup> In particular, they provide a detailed discussion on lexical and dynamic scoping.</p></li></ul> + +<h2 id="next-stop-r">Next stop, R</h2> + +<p>Now that we have examined the definitions of lexical and dynamic scope, and also explored some history, we are ready to return to the original question. <em>Is R lexically or dynamically scoped?</em></p> + +<p>In the <a href="/blog/2019/09/10/scoping-in-r/">next blog post</a>, we&rsquo;ll answer that question, and also see how R can be very confusing.</p> + +<p><em>I would like to thank Sam Caldwell, Ben Greenman, and Artem Pelenitsyn for their comments and feedback on this blog post.</em></p> + +<hr /> + +<div class="footnotes"> + <ol> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-1-definition" class="footnote-definition"> + <p>For example, at one point I defined lexical/dynamic scoping in terms of a &ldquo;lexical environment&rdquo; and a &ldquo;dynamic environment.&rdquo; But (1) that&rsquo;s a circular definition, (2) it assumes the reader has some intuition of how a &ldquo;lexical environment&rdquo; is different from a &ldquo;dynamic environment,&rdquo; and (3) it conflates two different kinds of &ldquo;environment.&rdquo;&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-1-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-2-definition" class="footnote-definition"> + <p>G. Steele. &ldquo;Scope and Extent,&rdquo; in <em>Common Lisp the Language</em>, 2nd ed. 1990. [<a href="https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html#SECTION00700000000000000000">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-2-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-3-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I,&rdquo; <em>Communications of the ACM</em>, vol. 3, no. 4, April 1960. [<a href="https://doi.org/10.1145/367177.367199">DOI</a>][<a href="http://jmc.stanford.edu/articles/recursive/recursive.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-3-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-4-definition" class="footnote-definition"> + <p>J. McCarthy. &ldquo;History of LISP,&rdquo; in <em>History of Programming Languages</em>, 1978. [<a href="https://doi.org/10.1145/800025.1198360">DOI</a>][<a href="http://jmc.stanford.edu/articles/lisp/lisp.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-4-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-5-definition" class="footnote-definition"> + <p>P. Naur (ed.). &ldquo;Revised Report on Algorithmic Language ALGOL 60,&rdquo; <em>Communications of the ACM</em>, vol. 6, no. 1, January 1963. [<a href="http://dx.doi.org/10.1145/366193.366201">DOI</a>][<a href="https://www.masswerk.at/algol60/report.htm">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-5-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-6-definition" class="footnote-definition"> + <p>P. Landin. &ldquo;The mechanical evaluation of expressions,&rdquo; <em>The Computer Journal</em>, vol. 6, no. 4, January 1964. [<a href="https://doi.org/10.1093/comjnl/6.4.308">DOI</a>][<a href="https://www.cs.cmu.edu/~crary/819-f09/Landin64.pdf">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-6-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-7-definition" class="footnote-definition"> + <p>J. Moses. &ldquo;The Function of FUNCTION in LISP or Why the FUNARG Problem Should be Called the Environment Problem,&rdquo; <em>SIGSAM Bulletin 15</em>, July 1970. [<a href="https://doi.org/10.1145/1093410.1093411">DOI</a>][<a href="https://dspace.mit.edu/handle/1721.1/5854">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-7-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-8-definition" class="footnote-definition"> + <p>G. Sussman and G. Steele. &ldquo;SCHEME: An Interpreter for Extended Lambda Calculus.&rdquo; 1975. [<a href="https://dspace.mit.edu/handle/1721.1/5794">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-8-return">↩</a></p></li> + <li id="2019-09-05-lexical-and-dynamic-scope-footnote-9-definition" class="footnote-definition"> + <p>G. Steele and G. Sussman. &ldquo;The Art of the Interpreter or, The Modularity Complex (Parts Zero, One, and Two).&rdquo; 1978. [<a href="https://dspace.mit.edu/handle/1721.1/6094">Available online</a>]&nbsp;<a href="#2019-09-05-lexical-and-dynamic-scope-footnote-9-return">↩</a></p></li></ol></div> \ No newline at end of file diff --git a/blog/feeds/semantics.atom.xml b/blog/feeds/semantics.atom.xml new file mode 100644 index 00000000..9fcb543b --- /dev/null +++ b/blog/feeds/semantics.atom.xml @@ -0,0 +1,315 @@ + + + PRL Blog: Posts tagged 'semantics' + + + urn:http-prl-ccs-neu-edu:-blog-tags-semantics-html + 2016-11-30T15:24:45Z + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-30-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages + 2016-11-30T15:24:45Z + 2016-11-30T15:24:45Z + + Jonathan Schuster + + + Beta Reduction (Part 1) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + 2016-11-02T21:10:18Z + 2016-11-02T21:10:18Z + + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/semantics.rss.xml b/blog/feeds/semantics.rss.xml new file mode 100644 index 00000000..3e94465e --- /dev/null +++ b/blog/feeds/semantics.rss.xml @@ -0,0 +1,313 @@ + + + + PRL Blog: Posts tagged 'semantics' + PRL Blog: Posts tagged 'semantics' + http://prl.ccs.neu.edu/blog/tags/semantics.html + Wed, 30 Nov 2016 15:24:45 UT + Wed, 30 Nov 2016 15:24:45 UT + 1800 + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + http://prl.ccs.neu.edu/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/?utm_source=semantics&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-30-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages + Wed, 30 Nov 2016 15:24:45 UT + Jonathan Schuster + + + Beta Reduction (Part 1) + http://prl.ccs.neu.edu/blog/2016/11/02/beta-reduction-part-1/?utm_source=semantics&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-02-beta-reduction-part-1 + Wed, 02 Nov 2016 21:10:18 UT + Milo Davis + +<p>The λ-calculus is often introduced by showing how to build a real programming language from it&rsquo;s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you&rsquo;re opening <a href="https://www.amazon.com/Calculus-Semantics-Studies-Foundations-Mathematics/dp/0444875085">Barendregt</a> for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).</p> +<!-- more--> + +<h1 id="introduction">Introduction</h1> + +<p>This post is aimed at myself around 18 months ago. At the time, I had spent a fair amount of time programming, taken a functional programming class, and been introduced to the λ-calculus. Despite that, I didn&rsquo;t really understand how the λ-calculus was really used in PL research and how it really worked. When I was asked to prove a novel theorem about β-reduction, I really struggled. I spent a lot of time looking online for an explanation of the proofs I could understand. This series of posts is my attempt to rectify that.</p> + +<p>This first post briefly introduces the λ-calculus and explains β-reduction through a formally defined semantics and examples in <a href="https://racket-lang.org/">Racket</a>, <a href="http://ocaml.org/">OCaml</a>, and <a href="https://www.haskell.org/">Haskell</a>. My goal in this post is to develop an intuition about what β-reduction is and how it works. In a followup post, I&rsquo;ll explain how to prove that β-reduction is confluent.</p> + +<h1 id="the-λ-calculus">The λ-Calculus</h1> + +<p>The λ-calculus is a simple model of computation developed by <a href="https://en.wikipedia.org/wiki/Alonzo_Church">Alonzo Church</a>. The λ-calculus has three different syntactic forms: variables, anonymous functions (lambdas), and function application. The <a href="http://matt.might.net/articles/grammars-bnf-ebnf/">BNF</a> for the λ-calculus is as follows:</p> + +<pre><code>e ::= x + | λx.e + | e e</code></pre> + +<p>In the above BNF, <code>x</code> is a metavariable, standing for any variable. In this post, I use <code>x</code>, <code>y</code>, <code>z</code>, <code>a</code>, and <code>b</code> as variables in my examples. The <code>λx.e</code> term represents a function with a single parameter <code>x</code>. We say that the parameter, <code>x</code> is bound in <code>e</code>. It is possible to have unbound variables in the λ-calculus, though for the most part, we can ignore them in our discussion of this topic. Finally, applications consist of two expressions. The first expression is the function and the second is its argument. Parenthesis are often added for clarity.</p> + +<p>If you program regularly in a functional language, this might look fairly familiar to you. In fact, you can compute anything in the λ-calculus that you can in any other language. In other words, the λ-calculus is Turing complete. Of course, you wouldn&rsquo;t want to program in this language as it&rsquo;s significantly harder to encode some of these constructs than just using built in language constructs like numbers. I&rsquo;m not going to discuss these ideas in detail, but if you&rsquo;re interested in how to add numbers, booleans, conditionals, and recursion to the λ-calculus, Matt Might has a few great posts about how to do this using equivalent constructs in <a href="http://matt.might.net/articles/python-church-y-combinator/">Python</a>, <a href="http://matt.might.net/articles/church-encodings-demo-in-scheme/">Scheme</a>, and <a href="http://matt.might.net/articles/js-church/">JavaScript</a>.</p> + +<p>Now that we&rsquo;ve discussed the syntax of the language, we can look at the semantics, or how terms evaluate. Below I have the evaluation rules for the λ-calculus. The arrow represents β-reduction which is the subject of this post. The semantics rely on substitution which is depicted with brackets. I have also defined the substitution function. I&rsquo;m assuming the Barendregt <a href="http://www.cse.chalmers.se/research/group/logic/TypesSS05/Extra/geuvers.pdf">variable convention (2.6)</a> which states that every bound variable is distinct from every free variable.</p> + +<pre><code>x[ x := e ] = e +y[ x := e ] = y +(λx.e1)[ x := e2 ] = (λx.e1[ x := e2 ]) +(e1 e2)[ x := e3 ] = (e1[ x := e3 ] e2[ x := e3 ])</code></pre> + +<p>With the substitution function defined, we can write a semantics for evaluation:</p> + +<pre><code>------------------------------ + (λx.e1) e2 -&gt;β e1[ x := e2 ] + + e1 -&gt;β e1' +-------------------- + e1 e2 -&gt;β e1' e2 + + e2 -&gt;β e2' +-------------------- + e1 e2 -&gt;β e1 e2'</code></pre> + +<p>These rules mean that if you have a term in which you have a function applied to an argument, you substitute the argument into the function. The next two rules say that if you can apply an evaluation rule to either the first or second subterm, you may do so. Notice that both the second and third rules might apply to a single term. These rules are nondeterministic and in these cases it is fine to use whichever rule you want (see below).</p> + +<h1 id="what-is-β-reduction">What is β-reduction?</h1> + +<p>More generally, what is reduction? Reduction is a model for computation that consists of a set of rules that determine how a term is stepped forwards. β-reduction is reduction by function application. When you β-reduce, you remove the λ from the function and substitute the argument for the function&rsquo;s parameter in its body. More formally, we can define β-reduction as follows:</p> + +<pre><code>(λx.e1) e2 = e1[ x := e2 ]</code></pre> + +<p>Given that definition, lets look at a few examples. I&rsquo;ve written the examples in the λ-calculus, Racket, OCaml, and Haskell. It&rsquo;s important to note that these languages have more restricted semantics than the original λ-calculus. These restrictions are called reduction strategies. In Racket and OCaml, it is not possible to substitute with anything except for a value, meaning you need to evaluate the argument until it is a function before substituting. This makes the evaluation reduce left to right before substituting. This restriction is called &ldquo;call by value&rdquo;. In Haskell, no reduction occurs in arguments, meaning that we would omit the third rule. This could potentially require an expression to be evaluated multiple times. In reality, Haskell caches the results of these evaluations so each expression is only evaluated once, but I&rsquo;m going to ignore that here. This presentation of Haskell&rsquo;s semantics is called &ldquo;call by name&rdquo; and the optimization is called &ldquo;call by need&rdquo;. (For more details on lazy evaluation, I recommend: <a href="http://www.ccs.neu.edu/racket/pubs/esop12-cf.pdf">Chang and Felleisen, ESOP 2012</a>.)</p> + +<h1 id="some-examples">Some Examples</h1> + +<p>The first example evaluates the same way in all of the languages.</p> + +<pre><code> (λx.x) (λy.y) +-&gt;β x[ x := (λy.y) ] += (λy.y)</code></pre> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="n">x</span><span class="p">[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="n">x</span><span class="o">[</span><span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="n">x</span><span class="p">[</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In our next example, we will see the differences between the languages. They all reduce to the same thing, albeit in different ways.</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)[</span><span class="n">x</span><span class="w"> </span><span class="kt">:=</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="p">))]</span><span class="w"></span> +<span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Note that the application on the right hand side is never evaluated. In an eager language like Racket or OCaml, the term being substituted must be a value, so the evaluation follows a different path.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="n">z</span><span class="p">[</span> <span class="n">z</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]))</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)[</span> <span class="n">x</span> <span class="n">:=</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="n">a</span><span class="p">)</span> <span class="p">]</span> +<span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3d))" style="color: inherit">=</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">((</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="n">z</span><span class="o">[</span> <span class="n">z</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">])</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)[</span> <span class="n">x</span> <span class="o">:=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">a</span><span class="o">)</span> <span class="o">]</span> +<span class="o">=</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The nondeterministic semantics of the λ-calculus lead to two possible paths through the above computation:</p> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λx.λy.y) (z[ z := (λa.a) ]) += (λx.λy.y) (λa.a) +-&gt;β (λy.y)[ x := (λa.a) ] += (λy.y)</code></pre> + +<pre><code> (λx.λy.y) ((λz.z) (λa.a)) +-&gt;β (λy.y)[ x := ((λz.z) (λa.a)) ] += (λy.y)</code></pre> + +<p>Lets look at a final example. This one is more complicated than the previous ones. I&rsquo;m also going to stop showing the substitution explicitly. In the literature, it is fairly common not to see it explicitly, but you should still be able to follow what&rsquo;s going on.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">(</span><span class="n">a</span> <span class="n">b</span><span class="p">)))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="n">b</span><span class="p">))</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">y</span><span class="p">)</span> <span class="n">y</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)))</span> +<span class="n">-&gt;β</span> <span class="p">((</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="n">x</span><span class="p">)</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">))</span> +<span class="n">-&gt;β</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">(</span><span class="n">z</span><span class="p">)</span> <span class="n">z</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The same thing happens in OCaml</p> + +<div class="brush: ocaml"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="n">a</span> <span class="n">b</span><span class="o">))</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">b</span> <span class="o">-&gt;</span> <span class="o">(</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="n">b</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">));;</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">((</span><span class="k">fun</span> <span class="n">y</span> <span class="o">-&gt;</span> <span class="n">y</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">))</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="n">x</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +<span class="o">-&gt;</span><span class="n">β</span> <span class="o">(</span><span class="k">fun</span> <span class="n">z</span> <span class="o">-&gt;</span> <span class="n">z</span><span class="o">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>In Haskell, the situation is a little different:</p> + +<div class="brush: haskell"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">x</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">))</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">a</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">b</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">y</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">y</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +<span class="ow">-&gt;</span><span class="n">β</span><span class="w"> </span><span class="p">(</span><span class="nf">\</span><span class="n">z</span><span class="w"> </span><span class="ow">-&gt;</span><span class="w"> </span><span class="n">z</span><span class="p">)</span><span class="w"></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, in the λ-calculus, things can go a few different ways.</p> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λx.x) ((λy.y) (λz.z)) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λa.λb.a b) (λy.y) (λz.z) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<pre><code> (λx.x) ((λa.λb.a b) (λy.y) (λz.z)) +-&gt;β (λx.x) ((λb.(λy.y) b) (λz.z)) +-&gt;β (λb.(λy.y) b) (λz.z) +-&gt;β (λy.y) (λz.z) +-&gt;β (λz.z)</code></pre> + +<p>There&rsquo;s a very interesting property of all of those examples: they all evaluate to the same thing, regardless of the reduction order taken. This is because β-reduction of the λ-calculus has the weak Church-Rosser Property. In systems that have this property, if a reduction sequence terminates, it will always evaluate to the same term, regardless of the path taken. This property does not necessarily hold if the term does not terminate. See if you can work out what happens to this term with each reduction strategy:</p> + +<pre><code>(λx.λy.y) ((λa.a a) (λb.b b))</code></pre> + +<p>A weaker property is called confluence, which states that all reduction sequences can be stepped to a common term. At the beginning of this post, I said that the λ-calculus is used as a model for real programming languages. This is generally true. There are proofs that the λ-calculus has this property, but people don&rsquo;t tend to prove such things about their actual languages, it is usually enough to build them knowing that their theoretical model has the property. In a follow up post, I&rsquo;ll explain the proofs that the λ-calculus does indeed have these properties.</p> + +<p>P.S. In Racket, you can look at the examples using the <a href="http://docs.racket-lang.org/stepper/">stepper</a> which will allow you to interactively run the term and see how it reduces.</p> \ No newline at end of file diff --git a/blog/feeds/statistics.atom.xml b/blog/feeds/statistics.atom.xml new file mode 100644 index 00000000..71e40e7b --- /dev/null +++ b/blog/feeds/statistics.atom.xml @@ -0,0 +1,133 @@ + + + PRL Blog: Posts tagged 'statistics' + + + urn:http-prl-ccs-neu-edu:-blog-tags-statistics-html + 2018-05-08T15:37:37Z + + Sampling Gradual Typing Performance + + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + 2018-05-08T15:37:37Z + 2018-05-08T15:37:37Z + Ben Greenman + Zeina Migeed + + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> \ No newline at end of file diff --git a/blog/feeds/statistics.rss.xml b/blog/feeds/statistics.rss.xml new file mode 100644 index 00000000..9e4749f0 --- /dev/null +++ b/blog/feeds/statistics.rss.xml @@ -0,0 +1,131 @@ + + + + PRL Blog: Posts tagged 'statistics' + PRL Blog: Posts tagged 'statistics' + http://prl.ccs.neu.edu/blog/tags/statistics.html + Tue, 08 May 2018 15:37:37 UT + Tue, 08 May 2018 15:37:37 UT + 1800 + + Sampling Gradual Typing Performance + http://prl.ccs.neu.edu/blog/2018/05/08/sampling-gradual-typing-performance/?utm_source=statistics&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-05-08-sampling-gradual-typing-performance + Tue, 08 May 2018 15:37:37 UT + Ben Greenman, Zeina Migeed + +<p>This post explains the sampling method introduced in the paper <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a></p> +<!-- more--> + +<h2 id="quick-reference-how-to-apply-the-method">Quick Reference: How to apply the method</h2> + +<ol> + <li>Find an untyped program, measure its running time.</li> + <li>Define a <em>granularity</em> for type annotations (by-function, by-module, by-program, &hellip;.).</li> + <li>Define a sample size <strong>s</strong> and number of samples <strong>r</strong>.</li> + <li>Randomly select <strong>s</strong> <em>configurations</em> uniformly at random, measure their running time.</li> + <li>Repeat the previous step <strong>r</strong> times.</li> + <li>Pick a positive real number <strong>D</strong>.</li> + <li>Count the proportion of configurations in each sample with running time less-than-or-equal-to <strong>D</strong></li> + <li>Build a 95% confidence interval for the <strong>r</strong> proportions computed in the previous step</li> + <li>Conclusion: there is a good chance that your interval contains the true proportion of configurations with running time less-than-or-equal-to <strong>D</strong></li></ol> + +<h2 id="background-what-to-measure">Background: what to measure</h2> + +<p>A migratory typing system adds static typing to a dynamically-typed (or, untyped) language. The recipe for &ldquo;adding static typing&rdquo; has a few steps:</p> + +<ul> + <li>add a syntax for type annotations</li> + <li>add a static type checker</li> + <li>add a semantics for statically-typed parts of the program</li></ul> + +<p>If the semantics for statically-typed parts of the program is <strong>not</strong> the same as the semantics for dynamically-typed parts, then it is important to measure performance.</p> + +<p>The key question is: how does adding type annotations affect the running time of a working program? We do not know how to answer this question directly.</p> + +<p>An easier question, that we can answer, is: for a few programs each with one full set of type annotations, how does adding or removing the chosen type annotations affect the running time of these programs?</p> + +<p>The next two sections give two methods for answering this question.</p> + +<h2 id="exhaustive-method">Exhaustive Method</h2> + +<p>One way to answer our easier question is to remove type annotations one &ldquo;unit&rdquo; at a time and measure the running time of all these partially-typed programs. We call the &ldquo;unit&rdquo; the <em>granularity</em> of the performance evaluation. For example, some choices for granularity are to remove types one module at a time, to remove types one function at a time, or to remove types one variable at a time. We call the &ldquo;partially-typed programs&rdquo; the <em>configurations</em> of the original dynamically-typed program. Note that the number of configurations depends on the choice of granularity &mdash; I can&rsquo;t just use the word <em>configurations</em> without telling you the granularity I have in mind.</p> + +<p>After measuring the running time of all configurations, we can summarize the results. One way to summarize is to pick a number <strong>D</strong> and count the number of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program. If this number is large, then the takeaway is: if <em>you</em> are willing to accept at most a <strong>D</strong>x slowdown, and you add your own type annotations to your own program, then there&rsquo;s some hope that your configuration runs at most <strong>D</strong> times slower than your original program.</p> + +<blockquote> + <p>Credit for the exhaustive method: <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> and <a href="https://www2.ccs.neu.edu/racket/pubs/ecoop2015-takikawa-et-al.pdf"><em>Toward Practical Gradual Typing</em></a></p></blockquote> + +<h2 id="simple-random-approximation-method">Simple Random Approximation Method</h2> + +<p>The method above does not scale to large programs or fine granularities because it asks for an exponential number of measurements. E.g., if there are 20 units to add or remove types from, then there are 1 million configurations to measure. Exponentials are bad.</p> + +<p><a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em></a>, suggests a method based on simple random sampling that answers a similar question. Instead of measuring the true proportion of configurations that run at most <strong>D</strong> times slower than the original dynamically-typed program, we:</p> + +<ul> + <li>pick a sample size <strong>s</strong> (in the paper, we used <strong>s = 10M</strong> where <strong>M</strong> is the number of units),</li> + <li>pick a number of samples <strong>r</strong> (in the paper, we used <strong>r = 10</strong>),</li> + <li>and build a 95% confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than the original program (from the <strong>r</strong> proportions of configurations that run at most <strong>D</strong> times slower than the original program in each of the <strong>r</strong> samples).</li></ul> + +<p>The method is outlined above, described in the paper, and validated in that paper&rsquo;s appendix. Please let us know if you have more questions.</p> + +<blockquote> + <p>Maybe you&rsquo;re wondering, &ldquo;gee why do they keep writing out &lsquo;configurations that run at most &hellip;.&rsquo; instead of something shorter?&rdquo;. Well, the short version is <em><strong>D</strong>-deliverable</em> and it was introduced in the <a href="https://www2.ccs.neu.edu/racket/pubs/popl16-tfgnvf.pdf"><em>Is Sound Gradual Typing Dead?</em></a> paper. Unfortunately, (1) that paper instantiated <strong>D</strong> to <strong>3</strong>-deliverable in order to explain a few graphs and (2) at least two published papers (<a href="https://dl.acm.org/citation.cfm?id=3009849">paper 1</a>, <a href="https://dl.acm.org/citation.cfm?id=3133878">paper 2</a>) now cite us as saying <strong>3</strong>x overhead is the cutoff between a good migratory typing system and a bad one.</p> + <p>&hellip;</p> + <p>If we can&rsquo;t trust scientists to understand, then we <em>definitely</em> can&rsquo;t trust you folks on the internet.</p></blockquote> + +<h2 id="faq">FAQ</h2> + +<h3 id="q-what-is-the-sampling-method-useful-for">Q. What is the sampling method useful for?</h3> + +<ul> + <li>Making a confidence interval for the true proportion of configurations that run at most <strong>D</strong> times slower than some baseline, for your favorite value of <strong>D</strong>.</li></ul> + +<h3 id="q-what-is-the-sampling-method-not-useful-for">Q. What is the sampling method <strong>not</strong> useful for?</h3> + +<ul> + <li>Finding the slowest configuration.</li> + <li>Finding the average running time of all configurations.</li> + <li>Evaluations where &ldquo;removing types&rdquo; might involve changing <strong>List[Int]</strong> to <strong>List[Dyn]</strong>, etc.</li> + <li>Situations where its wrong to assume that a programmer will start from untyped and pick a configuration uniformly at random</li> + <li>&hellip;. many more &hellip;.</li></ul> + +<h3 id="q-why-is-it-okay-to-choose-d-after-collecting-the-samples">Q. Why is it okay to choose <strong>D</strong> after collecting the samples?</h3> + +<p>The &ldquo;quick reference&rdquo; at the top of this post suggests choosing a value for <strong>D</strong> (the cutoff between good and bad performance) after sampling configurations and measuring their running time. This may sound strange, because (1) the value of <strong>D</strong> affects our bottom-line judgment about the proportion of configurations with good performance, and (2) shouldn&rsquo;t and value that affects the bottom line be fixed before taking samples? (To avoid accidental <a href="https://en.wikipedia.org/wiki/Data_dredging">data dredging</a>.)</p> + +<p>The reason it is ok to pick <strong>D</strong> after taking the sample is that the running times in the sample are independent of the choice of <strong>D</strong>.</p> + +<p>For example, if one person chose <strong>D=3</strong> and a second person chose <strong>D=9</strong>, both would follow the same protocol independent of <strong>D</strong> to take samples.</p> + +<h3 id="q-how-does-migratory-typing-relate-to-gradual-typing">Q. How does migratory typing relate to gradual typing?</h3> + +<p>Gradual typing is not just about adding a type system to an existing programming language. See <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/"><em>Refined Criteria for Gradual Typing</em></a> and <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/"><em>Migratory Typing: 10 Years Later</em></a> for details.</p> + +<h3 id="q-do-you-have-code-i-can-use-to-plot-sampling-data">Q. Do you have code I can use to plot sampling data?</h3> + +<p>Yes, start here:</p> + +<ul> + <li><a href="http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29">http://docs.racket-lang.org/gtp-plot/index.html#%28def._%28%28lib._gtp-plot%2Fplot..rkt%29._samples-plot%29%29</a></li></ul> + +<p>Please ask questions and open issues if you have trouble. The source is here:</p> + +<ul> + <li><a href="https://github.com/bennn/gtp-plot">https://github.com/bennn/gtp-plot</a></li></ul> + +<h3 id="q-where-is-code-for-the-sampling-paper">Q. Where is code for the sampling paper?</h3> + +<p>Start here:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018">https://pkgd.racket-lang.org/pkgn/package/gm-pepm-2018</a></li></ul> + +<p>Source is here:</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance">https://github.com/nuprl/retic_performance</a></li></ul> + +<h2 id="closing-thoughts">Closing Thoughts</h2> + +<p>Statistics is easy to do wrong. Please let us know if you think our method is doing bad statistics.</p> \ No newline at end of file diff --git a/blog/feeds/transient.atom.xml b/blog/feeds/transient.atom.xml new file mode 100644 index 00000000..4b85f45a --- /dev/null +++ b/blog/feeds/transient.atom.xml @@ -0,0 +1,1409 @@ + + + PRL Blog: Posts tagged 'transient' + + + urn:http-prl-ccs-neu-edu:-blog-tags-transient-html + 2020-11-12T10:15:16Z + + Transient for Optional and Keyword Functions + + urn:http-prl-ccs-neu-edu:-blog-2020-11-12-transient-for-optional-and-keyword-functions + 2020-11-12T10:15:16Z + 2020-11-12T10:15:16Z + + Ben Greenman + +<p>A short adventure into the depths of optional and/or keyword functions in Racket.</p> +<!-- more--> + +<hr /> + +<p>Transient, or rather <em>the Transient semantics for a mixed-typed language</em>, is one way to let statically-typed code safely interact with untyped code. You can read all about it in <a href="http://hdl.handle.net/2022/23172">Michael Vitousek&rsquo;s 2019 dissertation</a> or <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my 2020 dissertation</a>, and you can see how it compares to other mixed-typed semantics <a href="http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">here</a>. The idea is to give up on <a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">behavioral type guarantees</a> and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called <em>shape checks</em>; for example:</p> + +<ul> + <li>if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;</li> + <li>if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and</li> + <li>if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.</li></ul> + +<p>Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type <strong>T</strong> into a shape check, and we have a function with type <strong>T</strong> that needs to check its inputs. The question is how to actually do the check in Racket v7.9.</p> + +<p>In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if <strong>T = (-&gt; Symbol Boolean)</strong> then we need to check the shape <strong>symbol?</strong> of the domain type <strong>Symbol</strong>:</p> + +<pre><code>;; source code +(: f (-&gt; Symbol Boolean)) +(define (f sym) + (eq? sym 'hola)) + +;; ===&gt; + +;; imaginary (but realistic) rewritten code +(define (f sym) + (assert sym symbol?) + (eq? sym 'hola))</code></pre> + +<p>A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type <strong>T</strong> accepts 1 to 3 inputs:</p> + +<pre><code>;; source code +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) +(define (g #:a a [b 'b] #:c [c #f]) + (if a b (if c 'left 'right))) + +;; ===&gt; + +;; imaginary, unrealistic rewritten code +(define (g #:a a [b 'b] #:c [c #f]) + (assert a boolean?) + (assert b symbol?) + (assert c void?) + (if a b (if c 'left 'right)))</code></pre> + +<p>Good &mdash; we basically know what we want. If the Racket core language had optional and keyword functions, then we&rsquo;d be done.</p> + +<p>But no, Racket expands these optional/keyword functions into primitive <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._lam))"><strong>lambda</strong></a> and <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._case-lam))"><strong>case-lambda</strong></a> forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.</p> + +<p>Let&rsquo;s keep digging.</p> + +<p>From now on, &ldquo;Shallow&rdquo; or &ldquo;Shallow TR&rdquo; refers to my implementation of Transient for Typed Racket (TR). We&rsquo;ll talk about Shallow instead of &ldquo;Transient&rdquo; in case future work reveals a better way to implement the Transient idea.</p> + +<h2 id="false-start-follow-the-type">False Start: Follow the Type</h2> + +<p>Beware &mdash; Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function <strong>g</strong> above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional <strong>#:c</strong> keyword lead to a shape-check failure because the variable <strong>c</strong> gets the default value <strong>#f</strong> instead of a void value. Concretely, the third assert from above fails:</p> + +<pre><code>(define (g #:a a [b 'b] #:c [c #f]) + .... + (assert c void?) ;; fails if c is the #f default value + ....)</code></pre> + +<p>The problem arises from subtyping. According to the annotations, the function <strong>g</strong> has an external type that is less precise than the internal type that validates the function body:</p> + +<pre><code>;; external type T +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) + +;; internal type T2, subtype of external (T2 &lt;: T), validates body +(: g (-&gt;* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))</code></pre> + +<p>Thanks to this external / internal distinction, the following easy rewrite idea, <em>Solution 0</em>, fails. Despite the failure, this first solution is a useful starting point for a success.</p> + +<h4 id="solution-0-step-1-mimic-the-typechecker">Solution 0, Step 1: Mimic the Typechecker</h4> + +<p>Shallow TR uses the same type checker as classic <em>Deep</em> TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.</p> + +<p>Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.</p> + +<p>Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (<a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-expr-unit.rkt#L274-L295">typecheck/tc-expr-unit.rkt</a>). The omitted code (<strong>&hellip;.</strong>) does actual type checking:</p> + +<pre><code>(define (tc-expr/check/internal form expected-type) + .... + (syntax-parse form + #:literal-sets (kernel-literals tc-expr-literals) + .... + [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^) + ....] + [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>Ok! Those two patterns say a lot about the expansion of optional and keyword functions:</p> + +<ol> + <li>Both forms expand to a <strong>let-values</strong> that binds one function <strong>fun</strong>.</li> + <li>TR uses the syntax classes <strong>kw-lambda^</strong> and <strong>opt-lambda^</strong> to tell these particular <strong>let-values</strong> apart from others.</li></ol> + +<p>Shallow TR can use exactly these patterns to recognize optional/keyword functions.</p> + +<h4 id="solution-0-step-2-parse-the-domain-type">Solution 0, Step 2: Parse the Domain Type</h4> + +<p>Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function&rsquo;s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:</p> + +<pre><code> [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/type-table.rkt#L80"><strong>type-of</strong></a> function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, <strong>fun</strong>, has a type. Great.</p> + +<p>Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:</p> + +<ul> + <li>The type of <strong>fun</strong> could be a straightforward <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L693"><strong>Fun type</strong></a>, but it could also be a: <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L701"><strong>DepFun type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L521"><strong>Poly type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L531"><strong>PolyDots type</strong></a>, or even a <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L900"><strong>Union type</strong></a>.</li> + <li>Each part of the domain type corresponds to one parameter of the <strong>fun</strong> expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in <strong>fun</strong>, or the mandatory keywords?</li></ul> + +<h4 id="solution-0-step-3-insert-a-shape-check">Solution 0, Step 3: Insert a Shape Check</h4> + +<p>Once we have the target <strong>fun</strong> expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse <strong>fun</strong> to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting <strong>fun</strong> goes something like this:</p> + +<pre><code>(syntax-parse fun + [(#%plain-lambda formals . body) + #:with (shape-check ...) + (make-shape-checks #'formals (type-of fun)) + #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])</code></pre> + +<p>The rewritten function executes shape checks immediately, and then proceeds with the <strong>body</strong> after validating each actual parameter.</p> + +<h2 id="on-the-trail-optkey-expansion">On the Trail: optkey Expansion</h2> + +<p>Our <em>Solution 0</em> fails because the type of the <strong>fun</strong> expression that it gets from the type-checked code is an external type. In terms of the <strong>g</strong> function from above, <em>Solution 0</em> uses the type <strong>Void</strong> instead of the internal type <strong>(U Void #f)</strong> to check the <strong>c</strong> parameter. To get internal types, we need to look closer at <strong>fun</strong> and the rest of the optional/keyword expansion.</p> + +<p>Let&rsquo;s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.</p> + +<p>If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function <strong>X</strong> create a module <strong>test.rkt</strong> like this:</p> + +<pre><code>#lang racket/base + +(define _ignore + (let () + X + (void)))</code></pre> + +<p>Invoke the expander with <code>raco expand test.rkt &gt; test.rkt.txt</code> and explore the generated <strong>.txt</strong> file.</p> + +<h3 id="example-1-mandatory-keyword">Example 1: mandatory keyword</h3> + +<p>The source is a function with one mandatory positional argument and one optional positional argument.</p> + +<pre><code>(lambda (x [y 0]) + (+ x y))</code></pre> + +<p>Expansion generates a <strong>case-lambda</strong> that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function <strong>F</strong> that expects two arguments, resolves defaults in a different way, and executes the function body.</p> + +<pre><code>(let-values (((F) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F x '0)) + ((x y1) (#%app F x y1))))</code></pre> + +<p>Note: the expression <strong>(if &rsquo;#f &rsquo;0 y1)</strong> in the generated <strong>F</strong> function is equal to <strong>y1</strong> alone. In general, the <strong>if</strong> is for default expressions. (<a href="https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/">Unlike Python</a>, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a <strong>#f</strong> test. A general-purpose optimizer can remove this test before the code runs.</p> + +<h3 id="example-2">Example 2:</h3> + +<p>The source is a function with one mandatory positional argument and one mandatory keyword argument:</p> + +<pre><code>(lambda (x #:y y) + (+ x y))</code></pre> + +<p>Expansion generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> expects a list of keywords, a list of arguments, and a final argument. The purpose of <strong>F1</strong> is to organize a call to <strong>F0</strong>.</li> + <li><strong>lifted/2</strong> is the constructor for a generated struct type. Other functions help the struct call <strong>F1</strong>. Nevermind the details; I don&rsquo;t fully understand them either.</li></ul> + +<p>The important piece for Shallow TR is the <strong>F0</strong> function because the goal of rewriting is to protect the original function body against untyped inputs.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) y1)) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y1) (#%app car given-args))) + (#%app F0 y1 x3))))) + (#%app + lifted/2 + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l2571) given-kws)) + (if (#%app pair? l2571) + (if (#%app eq? (#%app car l2571) '#:y) + (#%app null? (#%app cdr l2571)) + '#f) + '#f)) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + '(#:y) + '(#:y))))</code></pre> + +<h3 id="example-3">Example 3:</h3> + +<p>The source is a function with one mandatory positional argument and one optional keyword argument:</p> + +<pre><code>(lambda (x #:y [y 0]) + (+ x y))</code></pre> + +<p>Expansion again generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments, resolves the optional default, and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> calls <strong>F0</strong></li> + <li>At the bottom, there are two <strong>case-lambda</strong> functions that call <strong>F1</strong></li></ul> + +<p>Again, the <strong>F0</strong> function is the focal point for Shallow TR rewriting.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y2) (#%app pair? given-kws))) + (let-values (((y1) + (if y2 (#%app car given-args) '0))) + (#%app F0 y1 x3)))))) + (#%app + make-optional-keyword-procedure + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l1571) given-kws)) + (let-values (((l1571) + (if (#%app null? l1571) + l1571 + (if (#%app eq? (#%app car l1571) '#:y) + (#%app cdr l1571) + l1571)))) + (#%app null? l1571))) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + null + '(#:y) + (case-lambda + ((x) (#%app F1 null null x))))))</code></pre> + +<h2 id="solution-the-shallow-tr-rewrite-strategy">Solution: The Shallow TR Rewrite Strategy</h2> + +<p>All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a <strong>let-values</strong> form:</p> + +<pre><code>(let-values (((f) fun)) . body)</code></pre> + +<p>Furthermore, the generated <strong>fun</strong> is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the <strong>fun</strong> from <em>Example 3</em> again; it has formal parameters for the keyword arg. and the mandatory arg., and one <strong>let-values</strong> to resolve each parameter:</p> + +<pre><code> (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y)))))</code></pre> + +<p>Another experiment with <strong>type-of</strong> shows that the right-hand side of each <strong>let-values</strong> has an internal type annotation. Excellent! Both <strong>(type-of x3)</strong> and <strong>(type-of (if &rsquo;#f &rsquo;0 y1))</strong> are the right types for shape checks. Shallow TR can:</p> + +<ul> + <li>inspect the <strong>let-values</strong> one-by-one;</li> + <li>convert the type of each right-hand expression to a shape predicate; and</li> + <li>rewrite each right-hand <strong>expr</strong> into <strong>(assert expr shape?)</strong>.</li></ul> + +<p>This should work! In fact, we can do slightly better:</p> + +<ul> + <li>when the right-hand expression is a conditional <strong>(if test default-expr supplied-arg)</strong></li> + <li>then Shallow only needs to check the supplied arg: <strong>(if test default-expr (assert supplied-arg shape?))</strong></li></ul> + +<p>Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.</p> + +<h2 id="a-problem-with-methods-and-a-bugfix">A Problem with Methods and a Bugfix</h2> + +<p>Currently, Shallow TR rewrites optional and keyword functions using the <strong>let-values</strong> plan described above. Each formal parameter has one <strong>let-values</strong> binding, and the type on each bound expression defines the shape check.</p> + +<p>Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we <a href="https://github.com/racket/racket/pull/3182">fixed the class expander</a>.</p> + +<p>First, here is a class with one method that runs correctly. The method <strong>f</strong> accepts an optional positional argument <strong>x</strong>; the default value of <strong>x</strong> is the current value of the field <strong>my-num</strong> (fields are mutable):</p> + +<pre><code>(define c0% + (class object% + (super-new) + (field (my-num 2)) + (define/public (f [x my-num]) + (+ x x))))</code></pre> + +<p>Second, here is a similar method that fails. This time, the default is an immediate value <strong>2</strong>:</p> + +<pre><code>(define c1% + (class object% + (super-new) + (define/public (f [x 2]) + (+ x x))))</code></pre> + +<p>Running a call <strong>(send o1 f)</strong> used to raise a shape-check failure about a strange value:</p> + +<blockquote> + <p>shape error: Expected a real number, got <code>#&lt;unsafe-undefined&gt;</code></p></blockquote> + +<p>What is going on?</p> + +<p>It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:</p> + +<pre><code>(lambda (x [y z]) + (+ x y))</code></pre> + +<p>Expansion generates a function <strong>F0</strong> that checks for the undefined value, and an outer <strong>case-lambda</strong> that supplies undefined when the default is needed:</p> + +<pre><code>(let-values (((F0) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) + (if (#%app eq? y1 unsafe-undefined) + z + y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F0 x unsafe-undefined)) + ((x y1) (#%app F0 x y1))))</code></pre> + +<p>That&rsquo;s the normal way that <strong>unsafe-undefined</strong> shows up: the <a href="https://github.com/racket/racket/blob/c0ff11e27bd28e070c20b7a9b0f7365f8f2b665a/racket/collects/racket/private/kw.rkt">expander for optional/keyword functions</a> looks for default expressions vs. default values and uses the undefined value for expressions.</p> + +<p>Three other facts conspired to make the problem with optional methods:</p> + +<ol> + <li>Typed Racket also looks for default expressions vs. default values (search for <strong>immediate-default</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/base-env/annotate-classes.rkt">here</a>). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the <strong>unsafe-undefined</strong> value (search for <strong>-Unsafe-Undefined</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/kw-types.rkt">here (kw)</a> and <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-lambda-unit.rkt">here (opt)</a>).</li> + <li>The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.</li> + <li>Shallow TR pushes default expression checks <strong>(if test default-expr supplied-arg)</strong> to the <strong>supplied-arg</strong> instead of wrapping the whole <strong>if</strong> form.</li></ol> + +<p>In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound&mdash;but harmless because the false assumption was guarded by an <strong>if</strong> test for <strong>unsafe-undefined</strong>. Running Shallow TR revealed the unsoundness with its eager shape check.</p> + +<p>Again, the resolution was to fix the class expander (<a href="https://github.com/racket/racket/pull/3182">racket/racket #3182</a>). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.</p> + +<h2 id="lessons">Lessons</h2> + +<ol> + <li>Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.</li> + <li>Digging into the expansion is sometimes necessary. There are at least three places that do so&mdash;the class expander, TR, and Shallow TR&mdash;and unfortunately they all need to cooperate.</li> + <li>The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my dissertation</a> lists them all.</li></ol> + + Transient Answers Old Questions + + urn:http-prl-ccs-neu-edu:-blog-2020-10-15-transient-answers-old-questions + 2020-10-15T13:32:12Z + 2020-10-15T13:32:12Z + + Ben Greenman + +<p>Several old questions from the Typed Racket mailing list have new and simple answers under a &ldquo;transient&rdquo; Typed Racket.</p> +<!-- more--> + +<hr /> + +<p>For the past few months, I&rsquo;ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the <a href="https://github.com/racket/typed-racket/pull/952">RFC</a> and <a href="https://github.com/racket/typed-racket/pull/948">pull request</a>.</p> + +<p>The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that&rsquo;s about it. By contrast, types are much stronger in classic Typed Racket.</p> + +<p>Shallow Racket&rsquo;s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the <a href="https://groups.google.com/g/racket-users">Racket mailing list</a>. There were a surprising number of questions that went like this:</p> + +<blockquote> + <p><strong>Q.</strong> Hey, I ran a program expecting <em>X</em> to happen, but <em>Y</em> happened instead. Is this a bug?</p> + <p><strong>A.</strong> No, Typed Racket has to do <em>Y</em> because of its strong types.</p></blockquote> + +<p>&hellip; but changing to shallow types gives the <em>X</em> behavior! Here are their stories.</p> + +<p>Going forward, <strong>Deep</strong> refers to normal Typed Racket and <strong>Shallow</strong> refers to Shallow Typed Racket.</p> + +<hr /> + +<h2 id="higher-order-value-as-any">Higher-Order Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ">groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ</a></p> + +<h4 id="on-20180416-mailoo-wrote">On 2018&ndash;04&ndash;16, <em>mailoo</em> wrote:</h4> + +<blockquote> + <p> I play a little with the &ldquo;Any&rdquo; type (due to &lsquo;dynamic-require&rsquo; which return Any), and I&rsquo;m not able to cast them back in a function.</p> + <p> I (over) simplify my question with this little program :</p></blockquote> + +<pre><code>(: p Any) +(define (p i) (displayln i)) + +; Here I want to get back my function +(define proc (cast p (-&gt; Integer Void))) +(proc 2) </code></pre> + +<blockquote> + <p> but I get this error when I try to execute the function :</p></blockquote> + +<pre><code>; contract violation +; Attempted to use a higher-order value passed as `Any` in untyped code: #&lt;procedure:p&gt; </code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> raises an error because it must enforce the <code>Any</code> type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> prints 2 and returns void. No error. Same goes for dynamic-require.</p> + +<hr /> + +<h2 id="parametric-contract-affects-untyped-code">Parametric Contract Affects Untyped Code</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ">groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ</a></p> + +<h4 id="on-20191215-john-clements-wrote">On 2019&ndash;12&ndash;15, John Clements wrote:</h4> + +<blockquote> + <p> It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:</p></blockquote> + +<pre><code> #lang typed/racket + + (require/typed racket/list + [index-of (All (T) ((Listof T) T -&gt; (U False Natural)))]) + + (index-of '(n s e w) 'n) ;; returns... #f? </code></pre> + +<blockquote> + <p> In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).</p> + <p> I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces parametricity for <code>All</code> types, and this throws off the equality function that index-of uses internally.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> returns 0.</p> + +<p>ps John, thanks very much for working on <a href="https://adventofcode.com">Advent of Code</a> and mailing the list!</p> + +<hr /> + +<h2 id="unable-to-protect-opaque-value-as-any">Unable to Protect Opaque Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ">groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ</a></p> + +<h4 id="on-20191211-marc-kaufmann-wrote">On 2019&ndash;12&ndash;11, Marc Kaufmann wrote:</h4> + +<blockquote> + <p>I have one file called <code>type-test.rkt</code> with the following</p></blockquote> + +<pre><code>#lang typed/racket + +(require (only-in typed/web-server/http response/xexpr response)) + +(provide f2) + +(: f2 (-&gt; (U response Any))) +(define (f2) + (define x '(body (h1 "Try it"))) + (: resp response) + (define resp (response/xexpr x)) + resp)</code></pre> + +<blockquote> + <p>Then I have another <em>untyped</em> file for a servlet:</p></blockquote> + +<pre><code>#lang racket + +(require "type-test.rkt" + web-server/servlet + web-server/servlet-env) + +(define (start req) + (f2)) + +(serve/servlet start + #:servlet-regexp #rx"" + #:launch-browser? #false + #:port 8080)</code></pre> + +<blockquote> + <p>Notice that I am telling [f2] that <code>resp</code> is of type <code>response</code>. Yet, when I run the server with <code>start</code> [&hellip;.] I get the following result:</p> + <p>(f2): Error, see below.</p> + <p>The error is:</p></blockquote> + +<pre><code>f2: broke its own contract + any-wrap/c: Unable to protect opaque value passed as `Any` + value: #&lt;response&gt; + in: the range of + (-&gt; Any)</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the <code>Any</code> type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.</p> + +<hr /> + +<h2 id="type-inference-installs-a-precise-type">Type Inference Installs a Precise Type</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ">groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ</a></p> + +<h4 id="on-20200214-john-clements-wrote">On 2020&ndash;02&ndash;14, John Clements wrote:</h4> + +<blockquote> + <p>I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.</p> + <p>Here’s a program:</p></blockquote> + +<pre><code>#lang typed/racket + +(define-type Store (Mutable-HashTable Integer Value)) +(define-type Value (U Real Boolean String)) + +(define top-store + (cast + (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f))) + Store)) + +(hash-set! top-store 5 1234)</code></pre> + +<blockquote> + <p>It fails with this error:</p></blockquote> + +<pre><code>contract violation +expected: (or/c (and/c byte? positive?) #t #f) +given: 1234 +in: the values of +the 3rd conjunct of +(and/c hash? + hash-mutable? + (hash/c exact-integer? + (or/c (and/c byte? positive?) #t #f) + #:immutable #f))</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p>First off, <strong>Deep</strong> runs fine after swapping <code>cast</code> for <code>ann</code>.</p> + +<p>Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.</p> + +<p>The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.</p> + +<hr /> + +<h2 id="same-arity-functions-in-a-case-lambda">Same-Arity Functions in a Case Lambda</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ">groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ</a></p> + +<h4 id="on-20190705-ryan-kramer-wrote">On 2019&ndash;07&ndash;05, Ryan Kramer wrote:</h4> + +<blockquote> + <p>In the code below, can <code>maybe-car</code> have the given type [&hellip;.]?</p></blockquote> + +<pre><code>#lang typed/racket + +(module untyped racket + (provide maybe-car) + (define (maybe-car x) + (cond + [(pair? x) (car x)] + [else x]))) + +(require/typed + 'untyped + [maybe-car (All (a b) (case-&gt; + (-&gt; (Pairof a b) a) + (-&gt; a a)))])</code></pre> + +<blockquote> + <p>[Current error:]</p></blockquote> + +<pre><code>Type Checker: + Type (All (a b) (case-&gt; (-&gt; (Pairof a b) a) (-&gt; a a))) + could not be converted to a contract: + function type has two cases of arity 1</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the type with a Racket <code>or/c</code> contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don&rsquo;t know which to apply to an incoming function.</p> + +<p>Note, the &ldquo;Type Checker&rdquo; error message is much better than what or/c would give on its own.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> simply checks that maybe-car accepts both arities inside the case-&gt; type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.</p> + +<hr /> + +<h3 id="immutable-type-affects-untyped-code">Immutable Type Affects Untyped Code</h3> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ">groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ</a></p> + +<h4 id="on-20200217-bertrand-augereau-wrote">On 2020&ndash;02&ndash;17, Bertrand Augereau wrote:</h4> + +<blockquote> + <p>Hello everybody, I&rsquo;m trying to gradually type my script to make it a proper app (yes I&rsquo;m a static-ish guy) and I have an issue (Racket 7.6 CS).</p></blockquote> + +<pre><code>; racket_mod.rkt: +#lang racket + +(provide (struct-out s)) +(provide list-of-s) +(provide set-list-of-s!) + +(struct s (a)) +(define list-of-s '()) +(define (set-list-of-s! los) + (set! list-of-s los))</code></pre> + +<pre><code>; racket_mod_typed.rkt: +#lang typed/racket + +(provide (struct-out s2)) +(provide list-of-s2) +(provide set-list-of-s2!) + +(struct s2 ([a : Natural])) +(define list-of-s2 '()) +(define (set-list-of-s2! [los : (Listof s2)]) + (set! list-of-s2 los))</code></pre> + +<pre><code>; racket_main.rkt: +#lang racket + +(require "racket_mod.rkt") +(require "racket_mod_typed.rkt") + +(define los (list (s 1) (s 2))) +(set-list-of-s! los) +(displayln list-of-s) + +(define los2 (list (s2 1) (s2 2))) +(set-list-of-s2! los2) +(displayln list-of-s2)</code></pre> + +<blockquote> + <p>list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces the type of <code>list-of-s2</code> with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.</p> + +<p>Here&rsquo;s a step-by-step breakdown:</p> + +<ol> + <li>the typed module creates an empty list-of-s2</li> + <li>the main module imports the list and receives a new copy</li> + <li>the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable</li> + <li>the main module reads from its copy &mdash; and it&rsquo;s still empty</li></ol> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> lets the original list travel to untyped code. There are no contracts in the way.</p> + +<h2 id="discussion">Discussion</h2> + +<p>Wow! It&rsquo;s great to see that Shallow Racket works &ldquo;as expected&rdquo; on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.</p> + +<p>If you have a similar experience with a deep-types error, let me know.</p> + +<p>Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error &mdash; and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.</p> + +<p>Shallow Typed Racket is coming soon. Follow the <a href="https://github.com/racket/typed-racket/pull/948">pull request</a> or watch the Racket release notes for news.</p> + +<h3 id="links">Links</h3> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">Larger example</a> where Shallow misses an error that Deep catches</li> + <li>Michael M. Vitousek <a href="http://hdl.handle.net/2022/23172">invented</a> the Transient semantics and implemented it in <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>.</li> + <li>My <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-thesis-2020">upcoming dissertation</a> has lots more to say about Shallow Typed Racket.</li></ul> + +<p><em>Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.</em></p> + + The Typed Racket Optimizer vs. Transient + + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + 2020-01-15T12:16:35Z + 2020-01-15T12:16:35Z + + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> + + Java and Migratory Typing + + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + 2018-12-02T14:41:53Z + 2018-12-02T14:41:53Z + + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> \ No newline at end of file diff --git a/blog/feeds/transient.rss.xml b/blog/feeds/transient.rss.xml new file mode 100644 index 00000000..781cf5ca --- /dev/null +++ b/blog/feeds/transient.rss.xml @@ -0,0 +1,1403 @@ + + + + PRL Blog: Posts tagged 'transient' + PRL Blog: Posts tagged 'transient' + http://prl.ccs.neu.edu/blog/tags/transient.html + Thu, 12 Nov 2020 10:15:16 UT + Thu, 12 Nov 2020 10:15:16 UT + 1800 + + Transient for Optional and Keyword Functions + http://prl.ccs.neu.edu/blog/2020/11/12/transient-for-optional-and-keyword-functions/?utm_source=transient&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-11-12-transient-for-optional-and-keyword-functions + Thu, 12 Nov 2020 10:15:16 UT + Ben Greenman + +<p>A short adventure into the depths of optional and/or keyword functions in Racket.</p> +<!-- more--> + +<hr /> + +<p>Transient, or rather <em>the Transient semantics for a mixed-typed language</em>, is one way to let statically-typed code safely interact with untyped code. You can read all about it in <a href="http://hdl.handle.net/2022/23172">Michael Vitousek&rsquo;s 2019 dissertation</a> or <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my 2020 dissertation</a>, and you can see how it compares to other mixed-typed semantics <a href="http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">here</a>. The idea is to give up on <a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">behavioral type guarantees</a> and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called <em>shape checks</em>; for example:</p> + +<ul> + <li>if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;</li> + <li>if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and</li> + <li>if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.</li></ul> + +<p>Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type <strong>T</strong> into a shape check, and we have a function with type <strong>T</strong> that needs to check its inputs. The question is how to actually do the check in Racket v7.9.</p> + +<p>In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if <strong>T = (-&gt; Symbol Boolean)</strong> then we need to check the shape <strong>symbol?</strong> of the domain type <strong>Symbol</strong>:</p> + +<pre><code>;; source code +(: f (-&gt; Symbol Boolean)) +(define (f sym) + (eq? sym 'hola)) + +;; ===&gt; + +;; imaginary (but realistic) rewritten code +(define (f sym) + (assert sym symbol?) + (eq? sym 'hola))</code></pre> + +<p>A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type <strong>T</strong> accepts 1 to 3 inputs:</p> + +<pre><code>;; source code +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) +(define (g #:a a [b 'b] #:c [c #f]) + (if a b (if c 'left 'right))) + +;; ===&gt; + +;; imaginary, unrealistic rewritten code +(define (g #:a a [b 'b] #:c [c #f]) + (assert a boolean?) + (assert b symbol?) + (assert c void?) + (if a b (if c 'left 'right)))</code></pre> + +<p>Good &mdash; we basically know what we want. If the Racket core language had optional and keyword functions, then we&rsquo;d be done.</p> + +<p>But no, Racket expands these optional/keyword functions into primitive <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._lam))"><strong>lambda</strong></a> and <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._case-lam))"><strong>case-lambda</strong></a> forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.</p> + +<p>Let&rsquo;s keep digging.</p> + +<p>From now on, &ldquo;Shallow&rdquo; or &ldquo;Shallow TR&rdquo; refers to my implementation of Transient for Typed Racket (TR). We&rsquo;ll talk about Shallow instead of &ldquo;Transient&rdquo; in case future work reveals a better way to implement the Transient idea.</p> + +<h2 id="false-start-follow-the-type">False Start: Follow the Type</h2> + +<p>Beware &mdash; Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function <strong>g</strong> above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional <strong>#:c</strong> keyword lead to a shape-check failure because the variable <strong>c</strong> gets the default value <strong>#f</strong> instead of a void value. Concretely, the third assert from above fails:</p> + +<pre><code>(define (g #:a a [b 'b] #:c [c #f]) + .... + (assert c void?) ;; fails if c is the #f default value + ....)</code></pre> + +<p>The problem arises from subtyping. According to the annotations, the function <strong>g</strong> has an external type that is less precise than the internal type that validates the function body:</p> + +<pre><code>;; external type T +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) + +;; internal type T2, subtype of external (T2 &lt;: T), validates body +(: g (-&gt;* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))</code></pre> + +<p>Thanks to this external / internal distinction, the following easy rewrite idea, <em>Solution 0</em>, fails. Despite the failure, this first solution is a useful starting point for a success.</p> + +<h4 id="solution-0-step-1-mimic-the-typechecker">Solution 0, Step 1: Mimic the Typechecker</h4> + +<p>Shallow TR uses the same type checker as classic <em>Deep</em> TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.</p> + +<p>Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.</p> + +<p>Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (<a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-expr-unit.rkt#L274-L295">typecheck/tc-expr-unit.rkt</a>). The omitted code (<strong>&hellip;.</strong>) does actual type checking:</p> + +<pre><code>(define (tc-expr/check/internal form expected-type) + .... + (syntax-parse form + #:literal-sets (kernel-literals tc-expr-literals) + .... + [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^) + ....] + [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>Ok! Those two patterns say a lot about the expansion of optional and keyword functions:</p> + +<ol> + <li>Both forms expand to a <strong>let-values</strong> that binds one function <strong>fun</strong>.</li> + <li>TR uses the syntax classes <strong>kw-lambda^</strong> and <strong>opt-lambda^</strong> to tell these particular <strong>let-values</strong> apart from others.</li></ol> + +<p>Shallow TR can use exactly these patterns to recognize optional/keyword functions.</p> + +<h4 id="solution-0-step-2-parse-the-domain-type">Solution 0, Step 2: Parse the Domain Type</h4> + +<p>Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function&rsquo;s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:</p> + +<pre><code> [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/type-table.rkt#L80"><strong>type-of</strong></a> function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, <strong>fun</strong>, has a type. Great.</p> + +<p>Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:</p> + +<ul> + <li>The type of <strong>fun</strong> could be a straightforward <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L693"><strong>Fun type</strong></a>, but it could also be a: <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L701"><strong>DepFun type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L521"><strong>Poly type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L531"><strong>PolyDots type</strong></a>, or even a <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L900"><strong>Union type</strong></a>.</li> + <li>Each part of the domain type corresponds to one parameter of the <strong>fun</strong> expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in <strong>fun</strong>, or the mandatory keywords?</li></ul> + +<h4 id="solution-0-step-3-insert-a-shape-check">Solution 0, Step 3: Insert a Shape Check</h4> + +<p>Once we have the target <strong>fun</strong> expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse <strong>fun</strong> to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting <strong>fun</strong> goes something like this:</p> + +<pre><code>(syntax-parse fun + [(#%plain-lambda formals . body) + #:with (shape-check ...) + (make-shape-checks #'formals (type-of fun)) + #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])</code></pre> + +<p>The rewritten function executes shape checks immediately, and then proceeds with the <strong>body</strong> after validating each actual parameter.</p> + +<h2 id="on-the-trail-optkey-expansion">On the Trail: optkey Expansion</h2> + +<p>Our <em>Solution 0</em> fails because the type of the <strong>fun</strong> expression that it gets from the type-checked code is an external type. In terms of the <strong>g</strong> function from above, <em>Solution 0</em> uses the type <strong>Void</strong> instead of the internal type <strong>(U Void #f)</strong> to check the <strong>c</strong> parameter. To get internal types, we need to look closer at <strong>fun</strong> and the rest of the optional/keyword expansion.</p> + +<p>Let&rsquo;s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.</p> + +<p>If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function <strong>X</strong> create a module <strong>test.rkt</strong> like this:</p> + +<pre><code>#lang racket/base + +(define _ignore + (let () + X + (void)))</code></pre> + +<p>Invoke the expander with <code>raco expand test.rkt &gt; test.rkt.txt</code> and explore the generated <strong>.txt</strong> file.</p> + +<h3 id="example-1-mandatory-keyword">Example 1: mandatory keyword</h3> + +<p>The source is a function with one mandatory positional argument and one optional positional argument.</p> + +<pre><code>(lambda (x [y 0]) + (+ x y))</code></pre> + +<p>Expansion generates a <strong>case-lambda</strong> that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function <strong>F</strong> that expects two arguments, resolves defaults in a different way, and executes the function body.</p> + +<pre><code>(let-values (((F) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F x '0)) + ((x y1) (#%app F x y1))))</code></pre> + +<p>Note: the expression <strong>(if &rsquo;#f &rsquo;0 y1)</strong> in the generated <strong>F</strong> function is equal to <strong>y1</strong> alone. In general, the <strong>if</strong> is for default expressions. (<a href="https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/">Unlike Python</a>, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a <strong>#f</strong> test. A general-purpose optimizer can remove this test before the code runs.</p> + +<h3 id="example-2">Example 2:</h3> + +<p>The source is a function with one mandatory positional argument and one mandatory keyword argument:</p> + +<pre><code>(lambda (x #:y y) + (+ x y))</code></pre> + +<p>Expansion generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> expects a list of keywords, a list of arguments, and a final argument. The purpose of <strong>F1</strong> is to organize a call to <strong>F0</strong>.</li> + <li><strong>lifted/2</strong> is the constructor for a generated struct type. Other functions help the struct call <strong>F1</strong>. Nevermind the details; I don&rsquo;t fully understand them either.</li></ul> + +<p>The important piece for Shallow TR is the <strong>F0</strong> function because the goal of rewriting is to protect the original function body against untyped inputs.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) y1)) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y1) (#%app car given-args))) + (#%app F0 y1 x3))))) + (#%app + lifted/2 + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l2571) given-kws)) + (if (#%app pair? l2571) + (if (#%app eq? (#%app car l2571) '#:y) + (#%app null? (#%app cdr l2571)) + '#f) + '#f)) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + '(#:y) + '(#:y))))</code></pre> + +<h3 id="example-3">Example 3:</h3> + +<p>The source is a function with one mandatory positional argument and one optional keyword argument:</p> + +<pre><code>(lambda (x #:y [y 0]) + (+ x y))</code></pre> + +<p>Expansion again generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments, resolves the optional default, and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> calls <strong>F0</strong></li> + <li>At the bottom, there are two <strong>case-lambda</strong> functions that call <strong>F1</strong></li></ul> + +<p>Again, the <strong>F0</strong> function is the focal point for Shallow TR rewriting.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y2) (#%app pair? given-kws))) + (let-values (((y1) + (if y2 (#%app car given-args) '0))) + (#%app F0 y1 x3)))))) + (#%app + make-optional-keyword-procedure + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l1571) given-kws)) + (let-values (((l1571) + (if (#%app null? l1571) + l1571 + (if (#%app eq? (#%app car l1571) '#:y) + (#%app cdr l1571) + l1571)))) + (#%app null? l1571))) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + null + '(#:y) + (case-lambda + ((x) (#%app F1 null null x))))))</code></pre> + +<h2 id="solution-the-shallow-tr-rewrite-strategy">Solution: The Shallow TR Rewrite Strategy</h2> + +<p>All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a <strong>let-values</strong> form:</p> + +<pre><code>(let-values (((f) fun)) . body)</code></pre> + +<p>Furthermore, the generated <strong>fun</strong> is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the <strong>fun</strong> from <em>Example 3</em> again; it has formal parameters for the keyword arg. and the mandatory arg., and one <strong>let-values</strong> to resolve each parameter:</p> + +<pre><code> (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y)))))</code></pre> + +<p>Another experiment with <strong>type-of</strong> shows that the right-hand side of each <strong>let-values</strong> has an internal type annotation. Excellent! Both <strong>(type-of x3)</strong> and <strong>(type-of (if &rsquo;#f &rsquo;0 y1))</strong> are the right types for shape checks. Shallow TR can:</p> + +<ul> + <li>inspect the <strong>let-values</strong> one-by-one;</li> + <li>convert the type of each right-hand expression to a shape predicate; and</li> + <li>rewrite each right-hand <strong>expr</strong> into <strong>(assert expr shape?)</strong>.</li></ul> + +<p>This should work! In fact, we can do slightly better:</p> + +<ul> + <li>when the right-hand expression is a conditional <strong>(if test default-expr supplied-arg)</strong></li> + <li>then Shallow only needs to check the supplied arg: <strong>(if test default-expr (assert supplied-arg shape?))</strong></li></ul> + +<p>Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.</p> + +<h2 id="a-problem-with-methods-and-a-bugfix">A Problem with Methods and a Bugfix</h2> + +<p>Currently, Shallow TR rewrites optional and keyword functions using the <strong>let-values</strong> plan described above. Each formal parameter has one <strong>let-values</strong> binding, and the type on each bound expression defines the shape check.</p> + +<p>Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we <a href="https://github.com/racket/racket/pull/3182">fixed the class expander</a>.</p> + +<p>First, here is a class with one method that runs correctly. The method <strong>f</strong> accepts an optional positional argument <strong>x</strong>; the default value of <strong>x</strong> is the current value of the field <strong>my-num</strong> (fields are mutable):</p> + +<pre><code>(define c0% + (class object% + (super-new) + (field (my-num 2)) + (define/public (f [x my-num]) + (+ x x))))</code></pre> + +<p>Second, here is a similar method that fails. This time, the default is an immediate value <strong>2</strong>:</p> + +<pre><code>(define c1% + (class object% + (super-new) + (define/public (f [x 2]) + (+ x x))))</code></pre> + +<p>Running a call <strong>(send o1 f)</strong> used to raise a shape-check failure about a strange value:</p> + +<blockquote> + <p>shape error: Expected a real number, got <code>#&lt;unsafe-undefined&gt;</code></p></blockquote> + +<p>What is going on?</p> + +<p>It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:</p> + +<pre><code>(lambda (x [y z]) + (+ x y))</code></pre> + +<p>Expansion generates a function <strong>F0</strong> that checks for the undefined value, and an outer <strong>case-lambda</strong> that supplies undefined when the default is needed:</p> + +<pre><code>(let-values (((F0) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) + (if (#%app eq? y1 unsafe-undefined) + z + y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F0 x unsafe-undefined)) + ((x y1) (#%app F0 x y1))))</code></pre> + +<p>That&rsquo;s the normal way that <strong>unsafe-undefined</strong> shows up: the <a href="https://github.com/racket/racket/blob/c0ff11e27bd28e070c20b7a9b0f7365f8f2b665a/racket/collects/racket/private/kw.rkt">expander for optional/keyword functions</a> looks for default expressions vs. default values and uses the undefined value for expressions.</p> + +<p>Three other facts conspired to make the problem with optional methods:</p> + +<ol> + <li>Typed Racket also looks for default expressions vs. default values (search for <strong>immediate-default</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/base-env/annotate-classes.rkt">here</a>). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the <strong>unsafe-undefined</strong> value (search for <strong>-Unsafe-Undefined</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/kw-types.rkt">here (kw)</a> and <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-lambda-unit.rkt">here (opt)</a>).</li> + <li>The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.</li> + <li>Shallow TR pushes default expression checks <strong>(if test default-expr supplied-arg)</strong> to the <strong>supplied-arg</strong> instead of wrapping the whole <strong>if</strong> form.</li></ol> + +<p>In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound&mdash;but harmless because the false assumption was guarded by an <strong>if</strong> test for <strong>unsafe-undefined</strong>. Running Shallow TR revealed the unsoundness with its eager shape check.</p> + +<p>Again, the resolution was to fix the class expander (<a href="https://github.com/racket/racket/pull/3182">racket/racket #3182</a>). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.</p> + +<h2 id="lessons">Lessons</h2> + +<ol> + <li>Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.</li> + <li>Digging into the expansion is sometimes necessary. There are at least three places that do so&mdash;the class expander, TR, and Shallow TR&mdash;and unfortunately they all need to cooperate.</li> + <li>The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my dissertation</a> lists them all.</li></ol> + + Transient Answers Old Questions + http://prl.ccs.neu.edu/blog/2020/10/15/transient-answers-old-questions/?utm_source=transient&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-10-15-transient-answers-old-questions + Thu, 15 Oct 2020 13:32:12 UT + Ben Greenman + +<p>Several old questions from the Typed Racket mailing list have new and simple answers under a &ldquo;transient&rdquo; Typed Racket.</p> +<!-- more--> + +<hr /> + +<p>For the past few months, I&rsquo;ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the <a href="https://github.com/racket/typed-racket/pull/952">RFC</a> and <a href="https://github.com/racket/typed-racket/pull/948">pull request</a>.</p> + +<p>The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that&rsquo;s about it. By contrast, types are much stronger in classic Typed Racket.</p> + +<p>Shallow Racket&rsquo;s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the <a href="https://groups.google.com/g/racket-users">Racket mailing list</a>. There were a surprising number of questions that went like this:</p> + +<blockquote> + <p><strong>Q.</strong> Hey, I ran a program expecting <em>X</em> to happen, but <em>Y</em> happened instead. Is this a bug?</p> + <p><strong>A.</strong> No, Typed Racket has to do <em>Y</em> because of its strong types.</p></blockquote> + +<p>&hellip; but changing to shallow types gives the <em>X</em> behavior! Here are their stories.</p> + +<p>Going forward, <strong>Deep</strong> refers to normal Typed Racket and <strong>Shallow</strong> refers to Shallow Typed Racket.</p> + +<hr /> + +<h2 id="higher-order-value-as-any">Higher-Order Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ">groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ</a></p> + +<h4 id="on-20180416-mailoo-wrote">On 2018&ndash;04&ndash;16, <em>mailoo</em> wrote:</h4> + +<blockquote> + <p> I play a little with the &ldquo;Any&rdquo; type (due to &lsquo;dynamic-require&rsquo; which return Any), and I&rsquo;m not able to cast them back in a function.</p> + <p> I (over) simplify my question with this little program :</p></blockquote> + +<pre><code>(: p Any) +(define (p i) (displayln i)) + +; Here I want to get back my function +(define proc (cast p (-&gt; Integer Void))) +(proc 2) </code></pre> + +<blockquote> + <p> but I get this error when I try to execute the function :</p></blockquote> + +<pre><code>; contract violation +; Attempted to use a higher-order value passed as `Any` in untyped code: #&lt;procedure:p&gt; </code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> raises an error because it must enforce the <code>Any</code> type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> prints 2 and returns void. No error. Same goes for dynamic-require.</p> + +<hr /> + +<h2 id="parametric-contract-affects-untyped-code">Parametric Contract Affects Untyped Code</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ">groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ</a></p> + +<h4 id="on-20191215-john-clements-wrote">On 2019&ndash;12&ndash;15, John Clements wrote:</h4> + +<blockquote> + <p> It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:</p></blockquote> + +<pre><code> #lang typed/racket + + (require/typed racket/list + [index-of (All (T) ((Listof T) T -&gt; (U False Natural)))]) + + (index-of '(n s e w) 'n) ;; returns... #f? </code></pre> + +<blockquote> + <p> In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).</p> + <p> I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces parametricity for <code>All</code> types, and this throws off the equality function that index-of uses internally.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> returns 0.</p> + +<p>ps John, thanks very much for working on <a href="https://adventofcode.com">Advent of Code</a> and mailing the list!</p> + +<hr /> + +<h2 id="unable-to-protect-opaque-value-as-any">Unable to Protect Opaque Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ">groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ</a></p> + +<h4 id="on-20191211-marc-kaufmann-wrote">On 2019&ndash;12&ndash;11, Marc Kaufmann wrote:</h4> + +<blockquote> + <p>I have one file called <code>type-test.rkt</code> with the following</p></blockquote> + +<pre><code>#lang typed/racket + +(require (only-in typed/web-server/http response/xexpr response)) + +(provide f2) + +(: f2 (-&gt; (U response Any))) +(define (f2) + (define x '(body (h1 "Try it"))) + (: resp response) + (define resp (response/xexpr x)) + resp)</code></pre> + +<blockquote> + <p>Then I have another <em>untyped</em> file for a servlet:</p></blockquote> + +<pre><code>#lang racket + +(require "type-test.rkt" + web-server/servlet + web-server/servlet-env) + +(define (start req) + (f2)) + +(serve/servlet start + #:servlet-regexp #rx"" + #:launch-browser? #false + #:port 8080)</code></pre> + +<blockquote> + <p>Notice that I am telling [f2] that <code>resp</code> is of type <code>response</code>. Yet, when I run the server with <code>start</code> [&hellip;.] I get the following result:</p> + <p>(f2): Error, see below.</p> + <p>The error is:</p></blockquote> + +<pre><code>f2: broke its own contract + any-wrap/c: Unable to protect opaque value passed as `Any` + value: #&lt;response&gt; + in: the range of + (-&gt; Any)</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the <code>Any</code> type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.</p> + +<hr /> + +<h2 id="type-inference-installs-a-precise-type">Type Inference Installs a Precise Type</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ">groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ</a></p> + +<h4 id="on-20200214-john-clements-wrote">On 2020&ndash;02&ndash;14, John Clements wrote:</h4> + +<blockquote> + <p>I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.</p> + <p>Here’s a program:</p></blockquote> + +<pre><code>#lang typed/racket + +(define-type Store (Mutable-HashTable Integer Value)) +(define-type Value (U Real Boolean String)) + +(define top-store + (cast + (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f))) + Store)) + +(hash-set! top-store 5 1234)</code></pre> + +<blockquote> + <p>It fails with this error:</p></blockquote> + +<pre><code>contract violation +expected: (or/c (and/c byte? positive?) #t #f) +given: 1234 +in: the values of +the 3rd conjunct of +(and/c hash? + hash-mutable? + (hash/c exact-integer? + (or/c (and/c byte? positive?) #t #f) + #:immutable #f))</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p>First off, <strong>Deep</strong> runs fine after swapping <code>cast</code> for <code>ann</code>.</p> + +<p>Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.</p> + +<p>The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.</p> + +<hr /> + +<h2 id="same-arity-functions-in-a-case-lambda">Same-Arity Functions in a Case Lambda</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ">groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ</a></p> + +<h4 id="on-20190705-ryan-kramer-wrote">On 2019&ndash;07&ndash;05, Ryan Kramer wrote:</h4> + +<blockquote> + <p>In the code below, can <code>maybe-car</code> have the given type [&hellip;.]?</p></blockquote> + +<pre><code>#lang typed/racket + +(module untyped racket + (provide maybe-car) + (define (maybe-car x) + (cond + [(pair? x) (car x)] + [else x]))) + +(require/typed + 'untyped + [maybe-car (All (a b) (case-&gt; + (-&gt; (Pairof a b) a) + (-&gt; a a)))])</code></pre> + +<blockquote> + <p>[Current error:]</p></blockquote> + +<pre><code>Type Checker: + Type (All (a b) (case-&gt; (-&gt; (Pairof a b) a) (-&gt; a a))) + could not be converted to a contract: + function type has two cases of arity 1</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the type with a Racket <code>or/c</code> contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don&rsquo;t know which to apply to an incoming function.</p> + +<p>Note, the &ldquo;Type Checker&rdquo; error message is much better than what or/c would give on its own.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> simply checks that maybe-car accepts both arities inside the case-&gt; type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.</p> + +<hr /> + +<h3 id="immutable-type-affects-untyped-code">Immutable Type Affects Untyped Code</h3> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ">groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ</a></p> + +<h4 id="on-20200217-bertrand-augereau-wrote">On 2020&ndash;02&ndash;17, Bertrand Augereau wrote:</h4> + +<blockquote> + <p>Hello everybody, I&rsquo;m trying to gradually type my script to make it a proper app (yes I&rsquo;m a static-ish guy) and I have an issue (Racket 7.6 CS).</p></blockquote> + +<pre><code>; racket_mod.rkt: +#lang racket + +(provide (struct-out s)) +(provide list-of-s) +(provide set-list-of-s!) + +(struct s (a)) +(define list-of-s '()) +(define (set-list-of-s! los) + (set! list-of-s los))</code></pre> + +<pre><code>; racket_mod_typed.rkt: +#lang typed/racket + +(provide (struct-out s2)) +(provide list-of-s2) +(provide set-list-of-s2!) + +(struct s2 ([a : Natural])) +(define list-of-s2 '()) +(define (set-list-of-s2! [los : (Listof s2)]) + (set! list-of-s2 los))</code></pre> + +<pre><code>; racket_main.rkt: +#lang racket + +(require "racket_mod.rkt") +(require "racket_mod_typed.rkt") + +(define los (list (s 1) (s 2))) +(set-list-of-s! los) +(displayln list-of-s) + +(define los2 (list (s2 1) (s2 2))) +(set-list-of-s2! los2) +(displayln list-of-s2)</code></pre> + +<blockquote> + <p>list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces the type of <code>list-of-s2</code> with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.</p> + +<p>Here&rsquo;s a step-by-step breakdown:</p> + +<ol> + <li>the typed module creates an empty list-of-s2</li> + <li>the main module imports the list and receives a new copy</li> + <li>the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable</li> + <li>the main module reads from its copy &mdash; and it&rsquo;s still empty</li></ol> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> lets the original list travel to untyped code. There are no contracts in the way.</p> + +<h2 id="discussion">Discussion</h2> + +<p>Wow! It&rsquo;s great to see that Shallow Racket works &ldquo;as expected&rdquo; on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.</p> + +<p>If you have a similar experience with a deep-types error, let me know.</p> + +<p>Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error &mdash; and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.</p> + +<p>Shallow Typed Racket is coming soon. Follow the <a href="https://github.com/racket/typed-racket/pull/948">pull request</a> or watch the Racket release notes for news.</p> + +<h3 id="links">Links</h3> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">Larger example</a> where Shallow misses an error that Deep catches</li> + <li>Michael M. Vitousek <a href="http://hdl.handle.net/2022/23172">invented</a> the Transient semantics and implemented it in <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>.</li> + <li>My <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-thesis-2020">upcoming dissertation</a> has lots more to say about Shallow Typed Racket.</li></ul> + +<p><em>Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.</em></p> + + The Typed Racket Optimizer vs. Transient + http://prl.ccs.neu.edu/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/?utm_source=transient&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + Wed, 15 Jan 2020 12:16:35 UT + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> + + Java and Migratory Typing + http://prl.ccs.neu.edu/blog/2018/12/02/java-and-migratory-typing/?utm_source=transient&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-12-02-java-and-migratory-typing + Sun, 02 Dec 2018 14:41:53 UT + Ben Greenman + +<p>The <em>transient</em> approach to migratory typing (circa <a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">2014</a>) is similar to type erasure in Java (circa <a href="https://docs.oracle.com/javase/1.5.0/docs/relnotes/features.html">2004</a>) in a few interesting ways.</p> +<!-- more--> + +<h2 id="migratory-typing">Migratory typing</h2> + +<p>The goal of <em>migratory typing</em> is to enrich the type system of a language without breaking backwards compatibility. Ideally, code that uses the enriched types:</p> + +<ul> + <li>(G1) benefits from new ahead-of-time checks,</li> + <li>(G2) benefits from stronger run-time guarantees, and</li> + <li>(G3) may interact with all kinds of existing code.</li></ul> + +<p>There are tradeoffs involved in the implementation of a migratory typing system, however, and (as we will see) different implementations may focus on different goals than the three above.</p> + +<p>A typical migratory typing system adds a static type checker to a dynamically typed language (<a href="/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/index.html">examples</a>), but one could also extend the type system of a statically-typed language; for example, by <a href="https://hal.inria.fr/hal-01629909v2">adding dependent types</a>. In this sense, Java 1.5.0 is a migratory typing system for pre-generics Java. The addition of generic types enabled new ahead-of-time checks and maintained backwards compatibility with existing Java code.</p> + +<p>Java&rsquo;s implementation of migratory typing has some interesting things in common with the <em>transient</em> implementation strategy recently proposed by Michael Vitousek and collaborators (<a href="http://homes.sice.indiana.edu/mvitouse/papers/dls14.pdf">DLS&rsquo;14</a>, <a href="https://mail.google.com/mail/u/0/h/1atrn21qlyrrh/?&amp;">POPL&rsquo;17</a>). The goal of this post is to demonstrate the connections.</p> + +<h2 id="erasure-migratory-typing">Erasure migratory typing</h2> + +<p>Before we compare Java 1.5.0 to transient, let&rsquo;s review a simpler strategy: the <em>erasure</em> approach to migratory typing.</p> + +<p><a href="https://www.typescriptlang.org/">TypeScript</a> is a great (modern) example of the erasure approach. TypeScript is a migratory typing system for JavaScript. A TypeScript module gets validated by an ahead-of-time type checker and compiles to JavaScript. After compilation, any JavaScript program may import bindings from the generated code. Conversely, a TypeScript module may import bindings from a JavaScript module by declaring a static type for each binding.</p> + +<blockquote> + <p>The <a href="http://definitelytyped.org/">DefinitelyTyped</a> repository provides TypeScript type definitions for many JavaScript libraries.</p></blockquote> + +<p>The TypeScript compiler erases types; every type <code>T</code> in the source code translates to the universal &ldquo;JavaScript type&rdquo;. For instance, a TypeScript function declaration compiles to an untyped JavaScript function:</p> + +<pre><code>(function (n0 : number, n1 : number) { return n0 + n1; }) + +// ==(compiles to)==&gt; + +(function (n0, n1) { return n0 + n1; })</code></pre> + +<p>TypeScript satisfies goals <strong>G1</strong> and <strong>G3</strong> for a migratory typing system because its type checker adds ahead-of-time checks and its compiler outputs JavaScript. TypeScript does not satisfy goal <strong>G2</strong> because the compiler erases types. In terms of the example above, the compiled function may be invoked with any pair of JavaScript values; the variable <code>n0</code> is not guaranteed to point to a <code>number</code> at run-time. On one hand, this means the type annotations have no effect on the behavior of a program &mdash; and in particular, cannot be trusted for debugging. On the other hand, it means that an experienced JavaScript programmer can re-use their knowledge to predict the behavior of a TypeScript program.</p> + +<p>In an ordinary program, the run-time guarantees of TypeScript are simply the run-time guarantees of JavaScript:</p> + +<ul> + <li>if a TypeScript expression <code>e</code> has the static type <code>T</code> and evaluates to a value <code>v</code>, then the only guarantee is that <code>v</code> is a valid JavaScript value (e.g., <code>T</code> could be <code>number</code> and <code>v</code> could be an incompatible object).</li></ul> + +<h2 id="transient-migratory-typing">Transient migratory typing</h2> + +<p><a href="https://github.com/mvitousek/reticulated">Reticulated</a> is a migratory typing system for Python that follows a <em>transient</em> implementation strategy. A Reticulated module gets type-checked and compiles to a Python module that defends itself from certain type-invalid inputs through the use of assertions that run in near-constant time. The type-checking addresses goal <strong>G1</strong>, the compilation to Python provides interoperability (goal <strong>G3</strong>), and the assertions partially meet goal <strong>G2</strong>.</p> + +<blockquote> + <p>These <em>certain</em> inputs are the ones that would cause a standard typed operational semantics to reach an undefined state. For a discussion of <em>near-constant</em>, see <a href="http://www.ccs.neu.edu/home/types/publications/publications.html#gm-pepm-2018"><em>On the Cost of Type-Tag Soundness</em>, section 2</a>.</p></blockquote> + +<p>For example, here is a Reticulated function that computes the average of a list of numbers:</p> + +<pre><code># Reticulated (commit e478343) +def average(nums : List(Float)) -&gt; Float: + if ns: + return sum(ns) / len(ns) + else: + raise ValueError("average: expected non-empty list")</code></pre> + +<p>and here is the Python code it compiles to:</p> + +<pre><code>from retic.runtime import * +from retic.transient import * +from retic.typing import * + +def average(nums): + check_type_list(nums) + if ns: + return check_type_float((check_type_function(sum)(ns) / check_type_function(len)(ns))) + else: + raise check_type_function(ValueError)('average: expected non-empty list')</code></pre> + +<blockquote> + <p>Note: the Reticulated syntax for type annotations is similar to the one proposed in <a href="https://www.python.org/dev/peps/pep-0484/">PEP 484</a>, but not identical. For example, Reticulated does not require forward references to be embedded in strings.</p></blockquote> + +<p>The Reticulated compiler removes all type annotations and inserts <code>check_type</code> assertions throughout the code. In <code>average</code>, these assertions check that: (1) the input is a list, (2) the output is a <code>float</code>, (3) and the names <code>sum</code> <code>len</code> and <code>ValueError</code> point to callable values. That&rsquo;s all. The assertions <strong>do not check</strong> that <code>nums</code> contains only floating-point numbers.</p> + +<blockquote> + <p>The assertions also do not check that the function bound to <code>sum</code> is defined for a single argument, which is arguably a bug. Scaling a model to an implementation is always challenging.</p></blockquote> + +<p>If <code>nums</code> contains something other than floating point numbers, then the call to <code>average</code> may cause <code>sum</code> to raise an exception or it may silently compute a nonsense result. The behavior depends on the implementation of <code>sum</code> in the same way that the behavior of a TypeScript function depends on any JavaScript functions that it invokes.</p> + +<p>Reticulated does not erase types, nor does it fully enforce types. Every type in a Reticulated module translates to its top-level type constructor <code>C(T)</code>, e.g.:</p> + +<pre><code> C(Float) = Float + C(List(Float)) = List + C(List(Float) -&gt; Float) = -&gt;</code></pre> + +<p>Consequently, Reticulated has a slightly stronger run-time guarantee than Python:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to have a top-level shape that matches the <code>C(T)</code> constructor.</li></ul> + +<h2 id="java-migratory-typing">Java migratory typing</h2> + +<p>Java 1.5.0 added <a href="https://www.jcp.org/en/jsr/detail?id=14">generic types</a> to the Java 1.4.0 type system. The benefit of generics is that a programmer can: write one class definition, use the definition in a few different contexts, and receive specific feedback from the type checker in each context.</p> + +<h3 id="review-generic-types">Review: generic types</h3> + +<p>Suppose we want to write a <code>Box</code> class that holds some kind of value; the value could be an <code>Integer</code> or a <code>String</code> or anything else. Here is a pre-generics definition:</p> + +<pre><code>class Box { + private Object val; + + public Box(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>With this definition is it possible to make boxes that hold different types of values:</p> + +<pre><code>// good! +Box iBox = new Box(new Integer(4)); +Box sBox = new Box(new String("X"));</code></pre> + +<p>but it is also possible to &ldquo;change the type&rdquo; of the contents of a <code>Box</code>:</p> + +<pre><code>// maybe bad! +iBox.set(new String("not a number"));</code></pre> + +<p>and some calls to <code>get</code> must be followed by a type cast:</p> + +<pre><code>// annoying! +((String) sBox.get()).charAt(0);</code></pre> + +<hr /> + +<p>With generics, we can give a name (e.g. <code>ValType</code>) to &ldquo;the type of the value inside a box&rdquo;:</p> + +<pre><code>class GBox&lt;ValType&gt; { + private ValType val; + + public GBox(ValType val) { this.set(val); } + + public void set(ValType val) { this.val = val; } + + public ValType get() { return this.val; } +}</code></pre> + +<p>and now we can tell the type checker to check different boxes differently (satisfying goal <strong>G1</strong>):</p> + +<pre><code>GBox&lt;Integer&gt; iBox = new GBox&lt;Integer&gt;(new Integer(0)); +GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + +// iBox.set(new String("not a number")); // Type Error, good! + +sBox.get().charAt(0); // no cast, good!</code></pre> + +<h3 id="backwards-compatibility--danger">Backwards compatibility &amp; danger</h3> + +<p>Java generics are backwards-compatible with older code (goal <strong>G3</strong>). This means that pre-generics code can interact with instances of a generic class. Vice-versa, generic code can interact with pre-generics classes. Since pre-generics code is not aware of type parameters, these interactions are potentially unsafe. For example, a pre-generics method can change the type of a <code>GBox</code>:</p> + +<pre><code>// Java 1.4.0 method +public static void evil(GBox b) { b.set(666); } + +// Java 1.5.0 method +public static void test() { + GBox&lt;String&gt; sBox = new GBox&lt;String&gt;(new String("A")); + evil(sBox); // OK, but generates unchecked warning + sBox.get().charAt(0); +}</code></pre> + +<p>The code above passes the type checker (with a warning about the <code>evil</code> method), and so it <em>seems</em> as though running the code will run the nonsense method call <code>666.charAt(0)</code> and lead to evil behavior. The actual result, however, is a cast error immediately after the call <code>sBox.get()</code> returns.</p> + +<p>Based on the cast error, we can tell that the compiler does not trust the type <code>GBox&lt;String&gt;</code> and inserts a run-time check that the result of the <code>.get()</code> is a string object.</p> + +<blockquote> + <p>&ldquo;Calling legacy code from generic code is inherently dangerous; once you mix generic code with non-generic legacy code, all the safety guarantees that the generic type system usually provides are void.&rdquo; <a href="https://www.oracle.com/technetwork/java/javase/generics-tutorial-159168.pdf">Generics in the Java Programming Language, Section 6.1</a></p></blockquote> + +<h3 id="java-type-erasure">Java Type Erasure</h3> + +<p>In order to support pre-generics and post-generics code on the same <a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">virtual machine</a>, the Java compiler <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.6">erases</a> generic type parameters after type-checking. Everywhere that the compiled code depends on an erased type, such as the <code>String</code> in <code>GBox&lt;String&gt;</code> above, Java adds a cast to prove to the Java Virtual Machine (JVM) that the erased bytecode is type-safe. (A smarter JVM type system might be able to prove that some casts are unnecessary via <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-thf.pdf">occurrence typing</a>.)</p> + +<p>After erasure, the <code>GBox&lt;ValType&gt;</code> class declaration loses its parameter:</p> + +<pre><code>// Erase `ValType`, replace with `Object` +class GBox { + private Object val; + + public GBox(Object val) { this.set(val); } + + public void set(Object val) { this.val = val; } + + public Object get() { return this.val; } +}</code></pre> + +<p>and the client code gains a cast:</p> + +<pre><code>GBox sBox = new GBox(new String("A")); + +((String) sBox.get()).charAt(0);</code></pre> + +<p>So far, so good. But it&rsquo;s worth noting that erasure can cause problems with Java arrays. An array needs to know the run-time type of its elements, so the following &ldquo;natural&rdquo; definition of an <code>ArrayList</code> is not permitted:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private T[] data; + private int size; + + public ArrayList(int capacity) { + data = new T[capacity]; + size = 0; + } + + public T get(int ix) { + // TODO bounds check + return data[ix] + } + + // .... +}</code></pre> + +<p>The trouble is that <code>T</code> does not say anything about the data that a new array needs to handle:</p> + +<pre><code>ArrayList.java:6: error: generic array creation + data = new T[capacity];</code></pre> + +<p>The only work-arounds require an array of objects and unchecked casts. One solution is to unsafely cast the array to the generic type:</p> + +<pre><code> // possibly dangerous, if `data` is aliased to an `Object[]` + public ArrayList(int capacity) { + data = (T[]) new Object[capacity]; + size = 0; + }</code></pre> + +<p>The other is to unsafely cast array elements in the <code>get</code> method, and elsewhere:</p> + +<pre><code>class ArrayList&lt;T&gt; { + private Object[] data; + private int size; + + public ArrayList(int capacity) { + data = new Object[capacity]; + size = 0; + } + + public T get(int ix) { + boundsCheck(ix); + return (T) data[ix]; + } + + // .... +}</code></pre> + +<p>Both may potentially lead to <a href="http://www.angelikalanger.com/GenericsFAQ/FAQSections/TechnicalDetails.html#FAQ050">heap pollution</a>.</p> + +<blockquote> + <p>"The decision not to make all generic types [not erased] is one of the most crucial, and controversial design decisions involving the type system of the Java programming language.</p> + <p>"Ultimately, the most important motivation for this decision is compatibility with existing code." <a href="https://docs.oracle.com/javase/specs/jls/se11/html/jls-4.html#jls-4.7">Java Language Specification, section 4.7</a></p></blockquote> + +<h3 id="run-time-guarantees">Run-time guarantees</h3> + +<p>By contrast to Reticulated&rsquo;s <code>C(T)</code> transformation, the following <code>G(T)</code> transformation describes generic-type erasure, where <code>T&lt;T1&gt;</code> describes a type <code>T</code> with parameter <code>T1</code> and <code>A[T1, T2]</code> describes a type variable <code>A</code> with lower bound <code>T1</code> and upper bound <code>T2</code>:</p> + +<pre><code> G(T&lt;T1&gt;) = G(T) + G(A[T1, T2]) = G(T1) + G(T) = T otherwise</code></pre> + +<p>If generic-type erasure results in a type mismatch (e.g., in <code>sBox.get().charAt(0)</code> above), the compiler inserts a cast. The inserted casts led to the run-time error in the previous example, and provide the following run-time guarantee:</p> + +<ul> + <li>if <code>e</code> is an expression with static type <code>T</code> that evaluates to a value <code>v</code>, then <code>v</code> is guaranteed to match the (bytecode) type <code>G(T)</code></li></ul> + +<h2 id="discussion">Discussion</h2> + +<p>TypeScript, Reticulated Python, and Java 1.5.0 each improved the type system of an existing language, but maintained backwards compatibility with existing code. The name <a href="http://drops.dagstuhl.de/opus/volltexte/2017/7120/">migratory typing</a> describes this kind of language extension.</p> + +<blockquote> + <p><a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">Gradual typing</a> is a similar; a gradual type system starts with a statically-typed language and adds dynamic typing in a principled way (<a href="https://pleiad.cl/papers/2016/garciaAl-popl2016.pdf">example</a>).</p></blockquote> + +<p>The TypeScript team had a choice between erasing types and enforcing types. They chose to erase types and run all code (typed or untyped) at the level of JavaScript. (Some TypeScript <a href="https://lorefnon.tech/2018/03/25/typescript-and-validations-at-runtime-boundaries/">libraries</a>, however, can enforce some types.)</p> + +<blockquote> + <p>TypeScript is not the only erasure language, nor is it the first. The oldest (I think) is <a href="http://www.softwarepreservation.org/projects/LISP/maclisp_family/">MACLISP</a>. For an erasure manifesto, see <a href="http://bracha.org/pluggableTypesPosition.pdf">Pluggable Type Systems</a>.</p></blockquote> + +<p>The Reticulated team faced an analogous choice, and chose to enforce the top-level shape of values in typed code (<a href="http://homes.sice.indiana.edu/mvitouse/papers/popl17.pdf">POPL 2017</a>). It will be interesting to see if this guarantee helps developers maintain programs, or if it is too shallow to be much use. The <a href="https://www.pyret.org/index.html">Pyret</a> language has been successful with comparable shallow checks.</p> + +<blockquote> + <p>Note: the POPL 2017 paper advertises an &ldquo;open-world soundness&rdquo;, but I do not see how this idea is different from the older idea of soundness in a multi-language system (<a href="https://www.eecs.northwestern.edu/~robby/pubs/papers/toplas09-mf.pdf">TOPLAS 2009</a>, <a href="https://www2.ccs.neu.edu/racket/pubs/dls06-tf.pdf">DLS 2006</a>).</p></blockquote> + +<p>Similarly, the Java team chose to erase generic types in Java 1.5.0 and use shallow casts in the JVM. The casts around type-erased generics provide a minimal level of safety &mdash; enough to prevent the use of a generic object from corrupting the state of a VM instance.</p> + +<p>Alternatively, Java could enforce full generic types at run-time. Over the years there have been a few proposals to do so (<a href="http://gafter.blogspot.com/2006/11/reified-generics-for-java.html">example 1</a>, <a href="https://wiki.openjdk.java.net/display/valhalla/Main">example 2</a>). The C# language has a similar type system and enforces generics at run-time (sources: <a href="https://mattwarren.org/2018/03/02/How-generics-were-added-to-.NET/">blog post</a>, <a href="https://www.microsoft.com/en-us/research/publication/design-and-implementation-of-generics-for-the-net-common-language-runtime/">PLDI 2001 paper</a>, <a href="https://dl.acm.org/citation.cfm?doid=378795.378797">backup link to paper</a>)</p> + +<h2 id="acknowledgments">Acknowledgments</h2> + +<p>Thank you to <a href="https://github.com/rmculpepper">Ryan Culpepper</a> and <a href="http://users.eecs.northwestern.edu/~jesse/">Jesse Tov</a> for noticing the similarity between Java&rsquo;s generic-type erasure and transient migratory typing. Jesse commented on an early version of this post, supplied new Java example code, and explained the trouble with generics and arrays.</p> \ No newline at end of file diff --git a/blog/feeds/turnstile.atom.xml b/blog/feeds/turnstile.atom.xml new file mode 100644 index 00000000..1552b39b --- /dev/null +++ b/blog/feeds/turnstile.atom.xml @@ -0,0 +1,746 @@ + + + PRL Blog: Posts tagged 'turnstile' + + + urn:http-prl-ccs-neu-edu:-blog-tags-turnstile-html + 2018-11-30T14:55:30Z + + Turnstile Mailing List + + urn:http-prl-ccs-neu-edu:-blog-2018-11-30-turnstile-mailing-list + 2018-11-30T14:55:30Z + 2018-11-30T14:55:30Z + + Stephen Chang + +<p><a href="https://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> now has a mailing list: <a href="https://groups.google.com/forum/#!forum/turnstile-users">https://groups.google.com/forum/#!forum/turnstile-users</a></p> +<!-- more--> + +<p>There is also a <code>#turnstile</code> channel on <a href="https://racket.slack.com">the Racket Slack</a>.</p> + + Defining Local Bindings in Turnstile Languages + + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + 2018-10-22T15:05:17Z + 2018-10-22T15:05:17Z + + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> \ No newline at end of file diff --git a/blog/feeds/turnstile.rss.xml b/blog/feeds/turnstile.rss.xml new file mode 100644 index 00000000..b33f310a --- /dev/null +++ b/blog/feeds/turnstile.rss.xml @@ -0,0 +1,744 @@ + + + + PRL Blog: Posts tagged 'turnstile' + PRL Blog: Posts tagged 'turnstile' + http://prl.ccs.neu.edu/blog/tags/turnstile.html + Fri, 30 Nov 2018 14:55:30 UT + Fri, 30 Nov 2018 14:55:30 UT + 1800 + + Turnstile Mailing List + http://prl.ccs.neu.edu/blog/2018/11/30/turnstile-mailing-list/?utm_source=turnstile&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-11-30-turnstile-mailing-list + Fri, 30 Nov 2018 14:55:30 UT + Stephen Chang + +<p><a href="https://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> now has a mailing list: <a href="https://groups.google.com/forum/#!forum/turnstile-users">https://groups.google.com/forum/#!forum/turnstile-users</a></p> +<!-- more--> + +<p>There is also a <code>#turnstile</code> channel on <a href="https://racket.slack.com">the Racket Slack</a>.</p> + + Defining Local Bindings in Turnstile Languages + http://prl.ccs.neu.edu/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/?utm_source=turnstile&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + Mon, 22 Oct 2018 15:05:17 UT + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> \ No newline at end of file diff --git a/blog/feeds/tutorial.atom.xml b/blog/feeds/tutorial.atom.xml new file mode 100644 index 00000000..7d2d845c --- /dev/null +++ b/blog/feeds/tutorial.atom.xml @@ -0,0 +1,5303 @@ + + + PRL Blog: Posts tagged 'tutorial' + + + urn:http-prl-ccs-neu-edu:-blog-tags-tutorial-html + 2019-02-17T16:20:50Z + + Writing a paper with Scribble + + urn:http-prl-ccs-neu-edu:-blog-2019-02-17-writing-a-paper-with-scribble + 2019-02-17T16:20:50Z + 2019-02-17T16:20:50Z + + Ben Greenman + +<p>This post explains how to get started using Scribble to write a research paper.</p> +<!-- more--> + +<hr /> + +<blockquote> + <p>This post was written using <a href="http://download.racket-lang.org/all-versions.html">Racket 7.1</a> and <a href="https://github.com/racket/scribble/releases/tag/v7.1">Scribble 1.29</a></p></blockquote> + +<p>Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:</p> + +<ul> + <li>the paper can import definitions from the research, keeping a single point of control;</li> + <li>the language&rsquo;s functional abstractions can help manage the writing;</li> + <li>the language&rsquo;s drawing and/or plotting libraries can replace <a href="https://ctan.org/pkg/pgf?lang=en">TikZ</a>;</li> + <li>and you can write unit tests to validate the claims made in the paper.</li></ul> + +<p>Scribble, <a href="http://docs.racket-lang.org/scribble/index.html">the Racket documentation tool</a>, comes with a to-LaTeX compiler and a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> library tailored to the new <a href="https://ctan.org/pkg/acmart?lang=en">ACM paper format</a>. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short &ldquo;getting started&rdquo; guide and links to some larger examples.</p> + +<blockquote> + <p>For a Scribble tutorial, see the links in: <a href="/blog/2017/05/23/building-a-website-with-scribble/index.html">Building a Website with Scribble</a></p></blockquote> + +<h2 id="getting-started-with-">Getting started with <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a></h2> + +<p>The first line of a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> document sets the formatting options (similar to a LaTeX file using <code>acmart.cls</code>). For example, the <a href="https://conf.researchr.org/track/gpce-2018/gpce-2018#Call-for-Papers">GPCE 2018 call for papers</a> asks for anonymized <code>sigplan</code>-format submissions with line numbers and 10 point font. The proper Scribble incantation is:</p> + +<pre><code>#lang scribble/acmart @sigplan @anonymous @review @10pt</code></pre> + +<p>Next, you may want to import some definitions. If we have a file <code>references.rkt</code> (see below for a definition), we can import it as follows:</p> + +<pre><code>@require{references.rkt}</code></pre> + +<p>The third main ingredient is the title and author information:</p> + +<pre><code>@(define neu (affiliation #:institution "Northeastern University")) +@(define anon (email "anon@anon.net")) + +@title{Writing a paper with Scribble} +@author[#:affiliation neu #:email anon]{Ben Greenman} + +@; optional: set the author names in the page headers +@elem[#:style "Sshortauthors"]{B. Greenman}</code></pre> + +<p>The paper is now ready to be written. You can forge ahead with a new <a href="http://docs.racket-lang.org/scribble/base.html#%28def._%28%28lib._scribble%2Fbase..rkt%29._section%29%29">section</a> and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and <a href="http://docs.racket-lang.org/scribble/base.html#%28form._%28%28lib._scribble%2Fbase..rkt%29._include-section%29%29">import</a> content from other modules:</p> + +<pre><code>@include-abstract{abstract.scrbl} +@include-section{introduction.scrbl}</code></pre> + +<p>Finally, the main page is a good place to <a href="https://docs.racket-lang.org/scriblib/autobib.html">generate the bibliography</a>. Assuming this document imports a file like the <code>references.rkt</code> below, this expression inserts a bibliography titled &ldquo;References&rdquo;:</p> + +<pre><code>@generate-bibliography[#:sec-title "References"]</code></pre> + +<p>To build the document, invoke <code>scribble</code> on the command-line with the <code>--pdf</code> or <code>--latex</code> options:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl</code></pre> + +<p>If all goes well, this command generates a <code>FILE.pdf</code> with properly-linked cross references.</p> + +<h3 id="auxiliary-files">Auxiliary Files</h3> + +<p>If you save the code above to a file <code>example.scrbl</code> and save the files below in the same directory, then you should be able to build an <code>example.pdf</code>.</p> + +<p>These files are available in a slightly different format at this link:</p> + +<ul> + <li><a href="https://gitlab.com/bengreenman/scribble-acmart-example">https://gitlab.com/bengreenman/scribble-acmart-example</a></li></ul> + +<h4 id="referencesrkt"><code>references.rkt</code></h4> + +<pre><code>#lang racket/base + +(provide + ~cite citet generate-bibliography + fbf-icfp-2009) + +(require + scriblib/autobib) + +(define-cite ~cite citet generate-bibliography + #:style author+date-square-bracket-style) + +(define icfp "ICFP") + +(define fbf-icfp-2009 + (make-bib + #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools" + #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler") + #:location (proceedings-location icfp #:pages '(109 120)) + #:date 2017))</code></pre> + +<h4 id="abstractscrbl"><code>abstract.scrbl</code></h4> + +<pre><code>#lang scribble/acmart + +A simple Scribble document.</code></pre> + +<h4 id="introductionscrbl"><code>introduction.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require{references.rkt} + +@; start with `title` instead of `section`, because importing via +@; `include-section` shifts all title/section/subsections down one level +@title{Introduction} + +Scribble creates a connection between a stand-alone document and the artifact +it describes@~cite[fbf-icfp-2009].</code></pre> + +<h3 id="q-how-to-debug-scribble-error-messages">Q. How to debug Scribble error messages?</h3> + +<p>If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.</p> + +<p>As a compile-time example, adding <code>@ foo</code> to a document produces the message <code>unexpected whitespace after @</code> and you can either delete the whitespace or change the <code>@</code> to <code>@"@"</code> for a literal <code>@</code>-sign.</p> + +<p>As a run-time example, adding <code>@(+ 2 2)</code> produces this message:</p> + +<pre><code>not valid in document body (need a pre-part for decode) in: 4</code></pre> + +<p>One fix is to convert <code>4</code> to a string, as in <code>@~a[(+ 2 2)]</code>.</p> + +<p>But if something goes wrong when Scribble renders a generated document to PDF, the default error output is <strong>not</strong> likely to help. For example, adding <code>@elem[#:style "oops"]</code> to a document produces a giant message:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl +[[ ... 84K of output ... ]] +Output written on example.pdf (1 page, 277876 bytes). +PDF statistics: + 53 PDF objects out of 1000 (max. 8388607) + 37 compressed objects within 1 object stream + 7 named destinations out of 1000 (max. 500000) + 36877 words of extra memory for PDF output out of 42996 (max. 10000000) + +run-pdflatex: got error exit code + context...: + [[ ... 17 more lines ... ]]</code></pre> + +<p>The best way to debug these messages is to <strong>ignore them</strong> and use a LaTeX compiler directly. For the &ldquo;oops&rdquo; mistake, LaTeX stops at the undefined control sequence &mdash; giving a hint about how to find the problem:</p> + +<pre><code>$ raco scribble --latex FILE.scrbl +$ pdflatex FILE.tex +[[ ... 12KB of output ... ]] +! Undefined control sequence. +l.549 \oops + {} +? </code></pre> + +<h3 id="q-how-to-add-a-latex-style-file">Q. How to add a LaTeX style file?</h3> + +<p>To add extra LaTeX code to the final document, create a new file and include it with the <code>++style</code> command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).</p> + +<pre><code>$ raco scribble ++style style.tex --pdf FILE.scrbl</code></pre> + +<p>Here is an example style file.</p> + +<h4 id="styletex"><code>style.tex</code></h4> + +<pre><code>\settopmatter{printfolios=true,printccs=true,printacmref=true} +% add page numbers etc. + +\overfullrule=1mm +% draw a black rectangle near lines that overflow the margin</code></pre> + +<p>Another way to add extra LaTeX code is to add a <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Flatex-properties..rkt%29._tex-addition%29%29"><code>tex-addition</code></a> style property to the main title. This second approach makes it easy to include more than one file:</p> + +<pre><code>#lang scribble/acmart + +@require[ + (only-in scribble/core make-style) + (only-in scribble/latex-properties make-tex-addition)] + +@(define extra-style-files + (list (make-tex-addition "style.tex"))) + +@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble} + +@; ....</code></pre> + +<h3 id="q-how-to-make-a-figure">Q. How to make a figure?</h3> + +<p>Use the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library to add figures to a document.</p> + +<pre><code>@require[pict scriblib/figure] +@figure[ + "fig:fish" @; figure tag, see `figure-ref` + @elem{A Standard Fish} @; figure caption, appears below the content + @elem{fish = @(standard-fish 90 40)}] @; content</code></pre> + +<p>The content of a figure can be almost anything that would work in the toplevel of the document.</p> + +<h3 id="q-how-to-include-extra-files-pictures-latex">Q. How to include extra files (pictures, LaTeX)?</h3> + +<p>The <code>++extra</code> command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.</p> + +<p>For example, if a document includes the content of an external LaTeX file:</p> + +<pre><code>@elem[#:style "input"]{inline-this.tex}</code></pre> + +<p>then make sure to build the document with a command like this:</p> + +<pre><code>$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl</code></pre> + +<h4 id="inline-thistex"><code>inline-this.tex</code></h4> + +<pre><code>% Raw LaTeX allowed here +$\lambda x.\, x$</code></pre> + +<h3 id="q-what-about-in-line-latex">Q. What about in-line LaTeX?</h3> + +<p>An <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Fcore..rkt%29._element%29%29">element</a> with the <a href="https://docs.racket-lang.org/scribble/core.html#%28idx._%28gentag._60._%28lib._scribblings%2Fscribble%2Fscribble..scrbl%29%29%29"><code>'exact-chars</code></a> <a href="https://docs.racket-lang.org/scribble/core.html#%28tech._style._property%29">style property</a> renders directly to LaTeX.</p> + +<pre><code>@(define (exact . stuff) + @; the style name "relax" puts a `\relax` no-op in front of the stuff + (make-element (make-style "relax" '(exact-chars)) stuff)) + +@exact|{$\lambda x.\, x$}| +@; ==&gt; \relax{$\lambda x.\, x$} + +@(define ($ . math-stuff) + (apply exact (list "$" math-stuff "$"))) + +@${\lambda x.\, x} +@; ==&gt; \relax{$\lambda x.\, x$}</code></pre> + +<h2 id="creating-a-httpdocsracket-langorgguidemodulesyntaxhtml28parthash-lang29lang-for-a-paper">Creating a <a href="http://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29">#lang</a> for a paper</h2> + +<p>For a Scribble document that is split across multiple files, it can be helpful to make a <code>#lang</code> that <a href="http://blog.racket-lang.org/2017/03/languages-as-dotfiles.html">provides a common environment</a>. Instead of starting each file with a <code>require</code>, e.g.:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require["references.rkt" "helper-functions.rkt" scriblib/figure] + +....</code></pre> + +<p>files can start with a name that describes their common purpose:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang conference-2018-submission + +....</code></pre> + +<p>As a bonus, if the language is defined as a package then the Scribble document can use Racket&rsquo;s dependency management tools:</p> + +<pre><code># to install the paper and interactively install dependencies: +$ cd conference-2018-submission; +$ raco pkg install + +# To check that the paper builds with no dependency issues: +$ raco setup --check-pkg-deps conference-2018-submission + +# To run all unit tests +$ raco test -c conference-2018-submission</code></pre> + +<p>To create a package and language:</p> + +<ol> + <li>Move the Scribble document to a directory with the language name, i.e., <code>conference-2018-submission/</code></li> + <li>Write a simple <code>info.rkt</code> to configure the package</li> + <li>Create a normal Racket module that exports the common environment</li> + <li>Create a <code>conference-2018-submission/lang/reader.rkt</code> module</li></ol> + +<p>Details below. For a full example, visit:</p> + +<ul> + <li><a href="https://gitlab.com/bennn/scribble-acmart-example">https://gitlab.com/bennn/scribble-acmart-example</a></li></ul> + +<hr /> + +<h4 id="conference-2018-submissioninforkt"><code>conference-2018-submission/info.rkt</code></h4> + +<p>This file defines the basic metadata for a package. For more about <code>info.rkt</code>, see: <a href="http://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html">Tutorial: Creating a Package</a>.</p> + +<pre><code>#lang info +(define collection "conference-2018-submission") +(define deps '("base" "scribble-lib" "at-exp-lib")) +(define build-deps '("racket-doc" "scribble-doc")) +(define pkg-desc "Paper for Conference 2018") +(define version "0.1")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionmainrkt"><code>conference-2018-submission/main.rkt</code></h4> + +<p>This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> language, the file &ldquo;references.rkt&rdquo;, and the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library.</p> + +<pre><code>#lang racket/base + +(provide + (all-from-out + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")) + +(require + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionlangreaderrkt"><code>conference-2018-submission/lang/reader.rkt</code></h4> + +<p>This file: (1) tells Racket to use the Scribble reader on <code>#lang conference-2018-submission</code> modules, and (2) wraps the result of such modules in a shape that Scribble expects.</p> + +<pre><code>#lang s-exp scribble/base/reader +conference-2018-submission +#:wrapper1 (lambda (t) (cons 'doc (t)))</code></pre> + +<h2 id="links-to-example-documents">Links to Example Documents</h2> + +<p>These documents use the <code>#lang</code> approach to writing a paper with Scribble. Check their <code>main.rkt</code> for example formatting functions and unit tests, and check the <code>.scrbl</code> files to see how the ideas above look in a larger document.</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018">https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018</a></li> + <li><a href="https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018">https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018</a></li></ul> + +<p>Finally, this repository provides a tool to start a new Scribble document:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gtp-paper">https://pkgd.racket-lang.org/pkgn/package/gtp-paper</a></li></ul> + +<h2 id="further-reading">Further Reading</h2> + +<ul> + <li><a href="https://project.inria.fr/coqexchange/checking-machine-checked-proofs/">Checking Machine-Checked Proofs</a></li></ul> + + Defining Local Bindings in Turnstile Languages + + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + 2018-10-22T15:05:17Z + 2018-10-22T15:05:17Z + + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> + + PLT Redex FAQ + + urn:http-prl-ccs-neu-edu:-blog-2017-09-25-plt-redex-faq + 2017-09-25T23:39:16Z + 2017-09-25T23:39:16Z + Ben Greenman + Sam Caldwell + + Ben Greenman, Sam Caldwell + +<p>A short guide to Redex concepts, conventions, and common mistakes.</p> +<!-- more--> + +<hr /> + +<p><em>To contribute to this FAQ, submit issues and pull requests to: <a href="https://github.com/nuprl/website/">https://github.com/nuprl/website/</a></em></p> + +<h3 id="q-what-is-redex-useful-for">Q. What is Redex useful for?</h3> + +<ol> + <li>declaring <a href="https://en.wikipedia.org/wiki/Regular_tree_grammar">regular tree grammars</a></li> + <li>defining <em>pattern</em>-based judgments and relations on <em>terms</em></li> + <li>testing properties of the above</li></ol> + +<p>More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.</p> + +<h3 id="q-what-is-redex-not-useful-for">Q. What is Redex <strong>not</strong> useful for?</h3> + +<p>Proving theorems about a grammar, judgment, or relation.</p> + +<h3 id="q-what-is-a-term">Q. What is a <em>term</em>?</h3> + +<p>Informally, a term is:</p> + +<ul> + <li>a Redex &ldquo;atom&rdquo;, or</li> + <li>an object that represents a sequence of characters.</li></ul> + +<p>More formally, a term is the result of evaluating <strong>(term X)</strong>, where <strong>X</strong> is any syntactically-correct Racket expression.</p> + +<p>Examples:</p> + +<pre><code>$ racket +Welcome to Racket v6.10.0.3. +&gt; (require redex/reduction-semantics) +&gt; (term 42) +42 +&gt; (term (+ 2 2)) +'(+ 2 2) +&gt; (term ("hello" world (#false))) +'("hello" world (#f))</code></pre> + +<p>Some terms may look strange. That&rsquo;s OK, because a term by itself has no meaning.</p> + +<p>Terms can refer to previously-defined values using the <strong>unquote</strong> escape.</p> + +<pre><code>&gt; (define x (term 42)) +&gt; (term (+ 2 x)) +'(+ 2 x) +&gt; (term (+ ,x (unquote x))) +'(+ 42 42)</code></pre> + +<h3 id="q-what-is-a-redex-model">Q. What is a <em>Redex model</em>?</h3> + +<p>A Redex model is collection of tools for working with terms. The tools may include:</p> + +<ul> + <li><em>languages</em>, to define a grammar for terms</li> + <li><em>judgments</em>, to describe properties of terms or relations between terms</li> + <li><em>metafunctions</em>, to transform terms</li> + <li><em>reduction relations</em>, to define a term rewriting system</li></ul> + +<p>The goal of these tools is to encode a &ldquo;real thing&rdquo; (maybe, a programming language) using Redex terms.</p> + +<h3 id="q-what-is-a-language">Q. What is a language?</h3> + +<p>A Redex <em>language</em> is a named set of non-terminals, <em>patterns</em>, and <em>binding forms</em>. For example, a Redex model of the natural numbers might start with this language:</p> + +<pre><code>(define-language nat + [N ::= Zero + (Plus1 N)])</code></pre> + +<ul> + <li>the name of the language is <strong>nat</strong></li> + <li>the non-terminal <strong>N</strong> is associated with two patterns: <strong>Zero</strong> and <strong>(Plus1 N)</strong></li> + <li>there are no <em>binding forms</em></li></ul> + +<p>Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.</p> + +<p>The non-terminal <strong>N</strong> describes all terms that are either:</p> + +<ol> + <li>the symbol <strong>Zero</strong></li> + <li>lists of the form <strong>(Plus1 N)</strong>, where <strong>N</strong> is either <strong>Zero</strong> or another &ldquo;Plus1&rdquo;</li></ol> + +<p>For example,</p> + +<pre><code>(term Zero) +(term (Plus1 Zero)) +(term (Plus1 (Plus1 Zero))) +(term (Plus1 (Plus1 (Plus1 Zero)))) +;; .... and so on</code></pre> + +<p>If a language has binding forms, then some terms can introduce names. See the question on <em>binding forms</em> (below) for an example.</p> + +<h3 id="q-what-is-a-pattern">Q. What is a pattern?</h3> + +<p>A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains <em>named non-terminals</em> from the language, then you can ask whether a Redex term matches the pattern.</p> + +<p>A <em>named non-terminal</em> for a language <strong>L</strong> is an identifier made of: (1) a non-terminal from <strong>L</strong>, (2) an underscore (<strong>_</strong>), and (3) any other identifier. See the FAQ entry below.</p> + +<p>For example, <strong>(redex-match? L p t)</strong> returns <strong>#true</strong> if the term <strong>t</strong> matches the pattern <strong>p</strong> relative to the language <strong>L</strong>.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(redex-match? nat N_some-name (term Zero)) +;; #true +(redex-match? nat (Plus1 N_a) (term Zero)) +;; #false +(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; #true</code></pre> + +<p>If <strong>(redex-match? L p t)</strong> is <strong>#true</strong>, then <strong>(redex-match L p t)</strong> shows how named non-terminals in the pattern bind to subterms of <strong>t</strong>.</p> + +<pre><code>(redex-match nat N_0 (term Zero)) +;; (list (match (list (bind 'N_0 'Zero)))) +(redex-match nat (Plus1 N_0) (term Zero)) +;; #f +(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))</code></pre> + +<h3 id="q-what-is-a-named-non-terminal">Q. What is a named non-terminal?</h3> + +<p>A named non-terminal in a language <strong>L</strong> is an identifier of the form <strong>X_Y</strong>, where:</p> + +<ul> + <li><strong>X</strong> is a non-terminal from <strong>L</strong></li> + <li><strong>Y</strong> is any identifier</li></ul> + +<p>The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.</p> + +<pre><code>(define-language trees + [binary-tree ::= Leaf + (Node binary-tree binary-tree)]) + +(redex-match trees + (Node binary-tree_left binary-tree_right) + (term (Node Leaf (Node Leaf Leaf)))) +;; (list +;; (match +;; (list (bind 'binary-tree_left 'Leaf) +;; (bind 'binary-tree_right '(Node Leaf Leaf)))))</code></pre> + +<h3 id="q-what-else-can-patterns-express">Q. What else can patterns express?</h3> + +<p>Redex patterns may contain special identifiers to guide pattern-matching. For instance:</p> + +<ul> + <li>The <strong>_</strong> pattern matches any term (and does not bind).</li> + <li>A pattern <strong>(p &hellip;)</strong> matches any sequence whose elements match the pattern <strong>p</strong>. If the pattern <strong>p</strong> is a named non-terminal, then the non-terminal binds a sequence of subterms.</li></ul> + +<p>Examples:</p> + +<pre><code>(redex-match? nat (Plus1 _) (term (Plus1 9))) +;; #true +(redex-match? nat (N_0 ...) (term ())) +;; #true +(redex-match? nat (N_0 ...) (term (Zero))) +;; #true +(redex-match nat (N_0 ...) (term (Zero Zero Zero))) +;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))</code></pre> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29">the Redex reference</a> for the full pattern language, including the <em>named and unique non-terminals</em> of the form <strong>X_!_Y</strong>.</p> + +<h3 id="q-what-can-patterns-not-express">Q. What can patterns <strong>not</strong> express?</h3> + +<ul> + <li>Disjunctions of patterns, e.g., &ldquo;number or boolean&rdquo;. (Use a language non-terminal.)</li> + <li>Negations of patterns. (Compose <strong>not</strong> with <strong>redex-match?</strong>.)</li> + <li>Some non-regular patterns. (Named dots <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29"><code>..._N</code></a> or <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-language%29%29"><code>define-language</code></a> with a side condition may be able to help.)</li></ul> + +<h3 id="q-what-is-a-judgment">Q. What is a judgment?</h3> + +<p>A Redex <em>judgment</em> form defines a relation on terms. The relation is defined by a set of inference rules.</p> + +<p>Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:</p> + +<ul> + <li>well-formedness conditions (i.e., whether a term contains free variables)</li> + <li>type checking rules</li> + <li>type inference rules</li> + <li>evaluation relations</li></ul> + +<p>Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(define-judgment-form nat + #:mode (N= I I) + #:contract (N= N N) + [ + --- Zero= + (N= Zero Zero)] + [ + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1) + (N= N_0-- N_1--) + --- Plus1= + (N= N_0 N_1)])</code></pre> + +<ol> + <li>the language is <strong>nat</strong>; Redex uses the language to interpret patterns</li> + <li>the mode is <strong>(N= I I)</strong>; this means <strong>N=</strong> is the name of a judgment that expects two input terms (or, <strong>N=</strong> is a binary relation on terms)</li> + <li>the contract is <strong>(N= N N)</strong>; in other words, <strong>N=</strong> expects two terms that match the <strong>N</strong> non-terminal from the <strong>nat</strong> language</li> + <li>there are two inference rules, named <strong>Zero=</strong> and <strong>Plus1=</strong></li> + <li>the <strong>Zero=</strong> rule states that <strong>(N= Zero Zero)</strong> always holds</li> + <li>the <strong>Plus1=</strong> rule states that <strong>(N= N_0 N_1)</strong> holds if <strong>N_0</strong> and <strong>N_1</strong> are both <strong>Plus1</strong> terms whose contents are related by <strong>N=</strong></li></ol> + +<p>The <strong>where</strong> clauses are <em>guards</em>. When Redex tries to apply a rule with premises of the form <strong>(where pattern term)</strong>, it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-judgment-form%29%29">the Redex reference</a> for more.</p> + +<pre><code>(judgment-holds (N= Zero Zero)) +;; #true +(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Note: the inference rules form a <em>set</em>, not a <em>sequence</em>. So when you ask Redex whether <strong>(judgment-holds (N= Zero Zero))</strong>, it applies all rules that match <strong>(N= Zero Zero)</strong>. For <strong>N=</strong> this is just one rule, but in general it could be many rules.</p> + +<h3 id="q-what-is-a-judgment-form-mode">Q. What is a judgment form <strong>#:mode</strong>?</h3> + +<p>A <strong>#:mode</strong> declaration expects a list of the form <strong>(id pos-use &hellip;)</strong>, where <strong>id</strong> is an identifier and each <strong>pos-use</strong> is either <strong>I</strong> or <strong>O</strong>. These declarations say four things:</p> + +<ol> + <li><strong>id</strong> is the name of a new judgment form</li> + <li><strong>id</strong> expects <strong>N</strong> arguments, where <strong>N</strong> is the number of <strong>pos-use</strong> symbols</li> + <li><strong>id</strong> expects an <em>input</em> at each position where the mode contains an <strong>I</strong></li> + <li><strong>id</strong> produces an <em>output</em> at each position where the mode contains an <strong>O</strong></li></ol> + +<p>For example, a type inference judgment may take an expression as input and output a type. Here&rsquo;s a fast and easy type inference judgment for arithmetic expressions. Given any term <strong>e_0</strong>, the judgment outputs the type <strong>Int</strong>.</p> + +<pre><code>(define-language Arith + (e ::= integer (e + e)) + (τ ::= Int)) + +(define-judgment-form Arith + #:mode (infer-type I O) + #:contract (infer-type e τ) + [ + --- T-Int + (infer-type e_0 Int)])</code></pre> + +<h3 id="q-what-can-judgments-not-express">Q. What can judgments <strong>not</strong> express?</h3> + +<p>Redex does not support inference rules that require guessing.</p> + +<p>One example of this is a transitivity rule: "<strong>τ_0</strong> is related to <strong>τ_2</strong> if there exists a <strong>τ_1</strong> such that <strong>τ_0</strong> is related to <strong>τ_1</strong> and <strong>τ_1</strong> is related to <strong>τ_2</strong>". The following example tries to define a transitive subtyping (<strong>&lt;:</strong>) relation, but Redex rejects the <strong>S-Trans</strong> rule.</p> + +<pre><code>(define-language SomeTypes + (τ ::= (→ τ τ) Integer)) + +(define-judgment-form SomeTypes + #:mode (&lt;: I I) + #:contract (&lt;: τ τ) + [ + (&lt;: τ_0 τ_1) + (&lt;: τ_1 τ_2) + --- S-Trans + (&lt;: τ_0 τ_2)] + [ + --- S-Refl + (&lt;: τ_0 τ_0)] + [ + (&lt;: τ_dom-1 τ_dom-0) + (&lt;: τ_cod-0 τ_cod-1) + --- S-Arrow + (&lt;: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])</code></pre> + +<p>The error is that in the rule <strong>S-Trans</strong>, the named non-terminal <strong>τ_1</strong> appears in an input position but is not bound to a term.</p> + +<h3 id="q-what-is-a-metafunction">Q. What is a metafunction?</h3> + +<p>A metafunction is a term-level function on terms.</p> + +<p>Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.</p> + +<p>Here is a metafunction that returns <strong>#true</strong> when given two equal natural numbers. The definition is similar to the <strong>N=</strong> judgment form.</p> + +<pre><code>(define-metafunction nat + N=? : N N -&gt; boolean + [(N=? Zero Zero) + #true] + [(N=? N_0 N_1) + (N=? N_0-- N_1--) + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1)] + [(N=? N_0 N_1) + #false])</code></pre> + +<ul> + <li>the metafunction is named <strong>N=?</strong></li> + <li>its contract is <strong>N N -&gt; boolean</strong>, this means <strong>N=?</strong> expects 2 terms that match the <strong>N</strong> pattern and returns a term that matches the pattern <strong>boolean</strong></li> + <li>there are three cases; the second case is guarded by two <strong>where</strong> clauses</li></ul> + +<p>Any occurrence of <strong>(N=? &hellip;.)</strong> in any term is evaluated.</p> + +<pre><code>(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(term ((N=? Zero Zero) Zero)) +;; '(#true Zero) +(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Any occurrence of <strong>N=?</strong> outside a <strong>term</strong> is an error.</p> + +<p>Metafunction <strong>where</strong>-clauses are analogous to judgment form <strong>where</strong>-clauses. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._metafunction%29">the Redex reference</a> for more.</p> + +<p>Note: the cases in a metafunction form a <em>sequence</em>, not a <em>set</em>. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.</p> + +<h3 id="q-should-i-use-a-metafunction-or-a-judgment-form">Q. Should I use a metafunction or a judgment form?</h3> + +<p>Use a judgment form.</p> + +<p>Metafunctions are handy, but judgments are easier to read and debug and maintain.</p> + +<h3 id="q-what-is-a-reduction-relation">Q. What is a reduction relation?</h3> + +<p>A reduction relation is a set of term-rewriting rules.</p> + +<p>Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.</p> + +<ul> + <li>The language tells Redex how to interpret patterns.</li> + <li>The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.</li> + <li>The rewrite rules have the form <strong>(&mdash;&gt; term term guard &hellip;)</strong>. The term on the left is the input, the term on the right is the output.</li></ul> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._reduction-relation%29%29">the Redex reference</a> for a full description of the guards.</p> + +<p>The preferred way to define a reduction relation is to define a language that includes three non-terminals:</p> + +<ol> + <li>a non-terminal for the domain of the reduction relation</li> + <li>a non-terminal for a <em>subset</em> of the domain that cannot reduce further</li> + <li>a non-terminal for evaluation contexts</li></ol> + +<p>An evaluation context is a term that contains a <strong>hole</strong>. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite &mdash; in particular, the sub-term that matches the <strong>hole</strong>.</p> + +<p>In the following example, <strong>bexp</strong> is the domain of a reduction relation. A <strong>bexp</strong> term represents a boolean expression, which can be <strong>#true</strong> or <strong>#false</strong> or a conjunction of expressions or a disjunction of expressions. The boolean expressions <strong>#true</strong> and <strong>#false</strong> are also values (<strong>val</strong>); these cannot reduce further. The non-terminal <strong>E</strong> is for evaluation contexts.</p> + +<pre><code>(define-language Bool + (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp)) + (val ::= #true #false) + (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E))) + +(define step + (reduction-relation Bool + #:domain bexp + [--&gt; (in-hole E (val_lhs ∧ val_rhs)) + (in-hole E val_new) + ∧-step + (where val_new ,(and (term val_lhs) (term val_rhs)))] + [--&gt; (in-hole E (val_lhs ∨ val_rhs)) + (in-hole E val_new) + ∨-step + (where val_new ,(or (term val_lhs) (term val_rhs)))]))</code></pre> + +<p>The function <strong>apply-reduction-relation</strong> applies a reduction relation to a term and returns a list of ways that the term can step.</p> + +<pre><code>(apply-reduction-relation step (term #true)) +;; '() +(apply-reduction-relation step (term (#true ∧ #true))) +;; '(#true) +(apply-reduction-relation step (term (#true ∧ #false))) +;; '(#false) +(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true))) +;; '((#true ∧ #true))</code></pre> + +<p>Three things about the reduction relation <strong>step</strong>:</p> + +<ol> + <li>Using <strong>in-hole</strong> on the first argument of <strong>&mdash;&gt;</strong> searches a term for a subterm that Redex can apply a reduction rule to.</li> + <li>Using <strong>in-hole</strong> on the second argument of <strong>&mdash;&gt;</strong> puts a new value back into the <strong>hole</strong> in the evaluation context.</li> + <li>The unquote operator (<strong>,</strong>) escapes to &ldquo;Racket mode&rdquo; (see below) to evaluate a conjunction or disjunction.</li></ol> + +<p>A judgment or metafunction is a formal alternative to &ldquo;escaping to Racket&rdquo;, but escaping can be convenient.</p> + +<p>Note: the cases in a reduction relation form a <em>set</em>, not a <em>sequence</em>. If more than one case matches, Redex applies them all.</p> + +<h3 id="q-what-is-racket-mode-what-is-redex-mode">Q. What is &ldquo;Racket mode&rdquo;? What is &ldquo;Redex mode&rdquo;?</h3> + +<p>Code in a Redex model is sometimes evaluated in &ldquo;Racket mode&rdquo; and sometimes evaluated in &ldquo;Redex mode&rdquo;. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.</p> + +<p>Key points:</p> + +<ol> + <li>A Redex program starts in Racket mode.</li> + <li>The <strong>X</strong> in <strong>(term X)</strong> is evaluated in Redex mode &hellip;</li> + <li>&hellip; unless <strong>X</strong> contains unquoted sub-expressions. Unquoting escapes to Racket mode &hellip;</li> + <li>&hellip; and <strong>term</strong>s inside an unquoted sub-expression are evaluated in Redex mode.</li></ol> + +<p>In other words, <strong>term</strong> enters Redex mode and <strong>unquote</strong> (<strong>,</strong>) escapes back to Racket.</p> + +<p>Redex implicitly switches to Redex mode in a few other places, for example:</p> + +<ul> + <li>the right-side of a <strong>where</strong> clause is in Redex mode</li> + <li>the result of a metafunction is in Redex mode</li></ul> + +<p>When in doubt, try using an <strong>unquote</strong>. Redex will raise an exception if it finds an unquote in Racket mode.</p> + +<h3 id="q-are-side-conditions-evaluated-in-racket-mode-or-redex-mode">Q. Are <strong>side-condition</strong>s evaluated in &ldquo;Racket mode&rdquo; or &ldquo;Redex mode&rdquo;?</h3> + +<p>A <strong>(side-condition e)</strong> sometimes evaluates <strong>e</strong> as a Racket expression and sometimes evaluates <strong>e</strong> as a Redex expression.</p> + +<ul> + <li>reduction relations and metafunctions expect a <strong>Racket</strong> expression</li> + <li>judgments expect a <strong>Redex</strong> expression</li></ul> + +<h3 id="q-what-is-a-binding-form">Q. What is a binding form?</h3> + +<p>In the lambda calculus, <strong>λ</strong>-terms bind variables. A term <strong>(λ x M)</strong> means that any free occurrence of <strong>x</strong> in the sub-term <strong>M</strong> refers to the <strong>x</strong> from the <strong>λ</strong>-term.</p> + +<p>Redex can express this idea with a binding form.</p> + +<pre><code>(define-language Λ + [e ::= (e e) x (λ x e)] + [x ::= variable-not-otherwise-mentioned] + #:binding-forms + (λ x_0 e_0 #:refers-to x_0))</code></pre> + +<p>Note: all the non-terminals in a language must be defined before the <strong>#:binding-forms</strong> keyword. If a non-terminal definition appears after the <strong>#:binding-forms</strong> keyword, then Redex will interpret the &ldquo;definition&rdquo; as a binding form.</p> + +<p>Binding forms work together with Redex&rsquo;s functions for substitution and alphabetic equivalence.</p> + +<pre><code>(alpha-equivalent? Λ + (term (λ x x)) + (term (λ y y)))) +;; #true + +(define-metafunction Λ + test-substitute : e -&gt; e + [(test-substitute (λ x_0 e_0)) + (substitute e_0 x_0 y)]) +(term (test-substitute (λ z (z z)))) +;; '(y y)</code></pre> + +<h3 id="q-what-is--what-is-">Q. What is <strong>&hellip;</strong>? What is <strong>&hellip;.</strong>?</h3> + +<p>Three dots (<strong>&hellip;</strong>) is for building patterns. If <strong>p</strong> is a pattern then <strong>(p &hellip;)</strong> matches any list whose elements all match <strong>p</strong>.</p> + +<pre><code>(define-language L) +(redex-match? L (number ... boolean ...) (term (1 2 #true #true))) +;; #true</code></pre> + +<p>Four dots (<strong>&hellip;.</strong>) may be used in <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-extended-language%29%29"><strong>define-extended-language</strong></a> to extend a previously-defined non-terminal.</p> + +<pre><code>(define-language C + (keyword ::= auto break case)) +(define-extended-language C++ + C + (keyword ::= .... class)) + +(redex-match? C keyword (term auto)) +;; #true +(redex-match? C keyword (term class)) +;; #false +(redex-match? C++ keyword (term auto)) +;; #true +(redex-match? C++ keyword (term class)) +;; #true</code></pre> + +<h3 id="q-where-to-learn-more-about-redex">Q. Where to learn more about Redex?</h3> + +<p>&ldquo;Critical path&rdquo; resources:</p> + +<ul> + <li>Code examples for this post: <a href="https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt">https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt</a></li> + <li>Redex documentation: <a href="http://docs.racket-lang.org/redex/index.html">http://docs.racket-lang.org/redex/index.html</a></li> + <li><a href="https://dvanhorn.github.io/redex-aam-tutorial/"><em>An Introduction to Redex with Abstracting Abstract Machines</em></a> by David Van Horn</li> + <li><a href="https://www.leafac.com/playing-the-game-with-plt-redex/"><em>Playing the Game with PLT Redex</em></a> by Leandro Facchinetti</li> + <li><a href="https://williamjbowman.com/doc/experimenting-with-redex/index.html"><em>Experimenting with Languages in Redex</em></a> by William J. Bowman</li></ul> + +<p>&ldquo;Procrastination&rdquo; resources:</p> + +<ul> + <li><a href="http://tata.gforge.inria.fr/"><em>Tree Automata Techniques and Applications</em></a></li> + <li><a href="http://lamport.azurewebsites.net/pubs/lamport-types.pdf"><em>Should your Specification Language be Typed?</em></a></li> + <li>Redex source code (see <code>redex-lib/</code>): <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + + Building a Website with Scribble + + urn:http-prl-ccs-neu-edu:-blog-2017-05-23-building-a-website-with-scribble + 2017-05-23T01:53:13Z + 2017-05-23T01:53:13Z + + Ben Greenman + +<p>The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!</p> +<!-- more--> + +<h2 id="the-story-so-far">The Story so Far</h2> + +<p>Last Fall, I took a flight to Chicago (on my way to <a href="http://con.racket-lang.org/2016/">RacketCon 2016</a>). When I landed, there was a new message in my inbox:</p> + +<pre><code> Subject: Web Page + Date: 2016-09-15 + + You have been nominated webmaster by public acclamation. Congratulations!</code></pre> + +<p>Emboldened by the trust of my people, I promptly converted the PRL website from Racket-generating-HTML to the fine <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> preprocessor language (commit <a href="https://github.com/nuprl/website/commit/a0600d32fec4bd70c5530b2717aec32979d634f7"><code>a0600d</code></a>) This bold action polarized the community.</p> + +<blockquote> + <p>I can&rsquo;t read the source anymore! Is this really an improvement?</p></blockquote> + +<p>Fear not, citizens. The switch to <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> was the right choice, and you too can learn to read the source code.</p> + +<h2 id="how-to-read-scribblehtml-programs">How to Read <code>scribble/html</code> Programs</h2> + +<h3 id="basics">Basics</h3> + +<p>Scribble is a language for writing Racket documentation. The key innovation in Scribble is the <em>@-expression</em> (read: &ldquo;at expression&rdquo;). The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language combines @-expression syntax with functions that generate HTML.</p> + +<h4 id="-syntax">@-syntax</h4> + +<p><a href="http://www.greghendershott.com/2015/08/at-expressions.html">Greg Hendershott</a> and the <a href="http://docs.racket-lang.org/scribble/reader.html">Scribble Documentation</a> explain @-expressions properly. Here&rsquo;s a short tutorial (Part 1 of 2, &ldquo;the basics&rdquo;):</p> + +<ul> + <li>Scribble programs start in &ldquo;text mode&rdquo;. Every character you type goes straight to the document you are building.</li> + <li>The @-sign toggles to &ldquo;Racket mode&rdquo; for the next expression. In Racket mode, the characters you type will be evaluated as a Racket program to produce part of the document.</li></ul> + +<p><em>Examples:</em> Evaluating <code>"Hello Dave"</code> puts &ldquo;Hello Dave&rdquo; in your document. Evaluating <code>"Hello @Dave"</code> puts &ldquo;Hello ???&rdquo; in your document, where "???" is the value of the variable <code>Dave</code>. Finally if <code>Dave</code> is the name of a function, then <code>"Hello @(Dave)"</code> calls the <code>Dave</code> function with zero arguments and puts whatever it returns into your document.</p> + +<p>To make it easy to interleave text, function calls, and code, Scribble discriminates between 4 kinds of parentheses when they follow an @-sign (Part 2 of 2, &ldquo;the parens&rdquo;):</p> + +<ul> + <li><code>@(f A B)</code> is just like the function call <code>(f A B)</code> in Racket</li> + <li><code>@f[A B]</code> is the same as <code>@(f A B)</code>, but typically more useful because &hellip;</li> + <li><code>@f[A B]{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; to a list of words <code>w*</code>, then calls <code>f</code> just like <code>(apply f A B w*)</code></li> + <li><code>@f{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; and calls <code>f</code> with the results</li> + <li><code>@f|{....}|</code> is similar, but the <code>....</code> are in &ldquo;unescapable text mode&rdquo;</li></ul> + +<p>&ldquo;Unescapable text mode&rdquo; treats @-signs as text instead of toggling between modes.</p> + +<h4 id="generating-html">Generating HTML</h4> + +<p>The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language comes with functions that render HTML. These functions have the same name as the corresponding HTML tag.</p> + +<p>Example program:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this program prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>No surprises.</p> + +<p>One thing that <em>is</em> surprising is how <code>scribble/html</code> handles tag attributes. Every tag-rendering function accepts &ldquo;Racket mode&rdquo; arguments that specify an attribute name and attribute value.</p> + +<p>For example:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">[</span><span class="n">style:</span> <span class="s2">"color:red"</span><span class="p">]{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Hope the output looks familiar. The input syntax is strange, but that&rsquo;s what it is.</p> + +<p>Larger programs print larger webpages. Each page on the PRL website is HTML generated by one <code>scribble/html</code> program.</p> + +<h2 id="why-scribblehtml-is-an-improvement">Why <code>scribble/html</code> is an Improvement</h2> + +<p>Before <code>scribble/html</code>, the PRL website was implemented in <code>scribble/text</code>. A <code>scribble/text</code> program renders and prints text. There is no extra support for HTML.</p> + +<p>To compare, here&rsquo;s the start of the old homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/text</span> +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="s2">"templates.rkt"</span><span class="p">)</span> + +<span class="n">&lt;!DOCTYPE</span> <span class="n">html&gt;</span> +<span class="n">&lt;html</span> <span class="n">lang=</span><span class="s2">"en"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">header</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;body</span> <span class="n">id=</span><span class="s2">"pn-top"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">navbar</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;div</span> <span class="n">class=</span><span class="s2">"jumbotron"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And here is the start of the <code>scribble/html</code>&rsquo;d homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@require</span><span class="p">[</span><span class="s2">"templates.rkt"</span><span class="p">]</span> + +<span class="n">@doctype</span><span class="p">{</span><span class="n">html</span><span class="p">}</span> +<span class="n">@html</span><span class="p">[</span><span class="n">lang:</span> <span class="s2">"en"</span><span class="p">]{</span> + <span class="n">@header</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@body</span><span class="p">[</span><span class="n">id:</span> <span class="s2">"pn-top"</span><span class="p">]{</span> + <span class="n">@navbar</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@div</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"jumbotron"</span><span class="p">]{</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The pages look similar. The new one has more @-signs and parentheses, the old one has more <code>&lt;</code>-signs and quotes. If you were able to edit the old page, you should be able to edit the new page.</p> + +<p>The <strong>key improvement</strong> in the new page is that <strong>common mistakes are now compile-time errors</strong>.</p> + +<ul> + <li> + <p>Before, a typo like <code>&lt;hmtl&gt;</code> would generate an ugly webpage. After, a typo like <code>@hmtl</code> is a syntax error.</p></li> + <li> + <p>Before, a typo like <code>&lt;b&gt;....</code> with no closing tag would generate an ugly webpage. After, a typo like <code>@b{....</code> is a syntax error.</p></li></ul> + +<p>Both flavors of error message come with source-code line numbers. This is very very helpful.</p> + +<h3 id="small-improvements">Small Improvements</h3> + +<h4 id="1-more-functions">1. More Functions</h4> + +<p>Before, the <a href="http://prl.ccs.neu.edu/teaching.html">Teaching page</a> contained some interesting HTML for rendering vertical text (look for the word &ldquo;Semantics&rdquo; to see how this was used):</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"how-to-design-programs"</span><span class="p">&gt;</span>S<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>e<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>m<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>a<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>n<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>t<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>i<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>c<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>s<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>After, the same text is generated from a function call:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@span</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"how-to-design-programs"</span><span class="p">]{</span><span class="n">@vertical-text</span><span class="p">{</span><span class="n">Semantics</span><span class="p">}}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The <code>vertical-text</code> function is simple:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@require</span><span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._only-in))" style="color: inherit">only-in</a></span> <span class="n">racket/list</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span><span class="p">)]</span> + +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">vertical-text</span> <span class="o">.</span> <span class="n">str*</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-~3elist))" style="color: inherit">string-&gt;list</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._append*))" style="color: inherit">append*</a></span> <span class="n">str*</span><span class="p">))</span> <span class="p">(</span><span class="n">br</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="2-more-structure-less-boilerplate">2. More Structure, Less Boilerplate</h4> + +<p>Here&rsquo;s part of the old definition of &ldquo;Ben Greenman&rdquo; on the <a href="http://prl.ccs.neu.edu/people.html">People page</a>:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"row pn-person"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-row-eq-height"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-photo"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"img-wrapper"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">"img/ben_greenman.jpg"</span> <span class="na">title</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="na">alt</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="p">/&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-9"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-4 pn-contact"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"pn-name"</span><span class="p">&gt;</span>Ben Greenman<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + Advisor: Matthias Felleisen<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"mailto:types@"</span><span class="err">@"</span><span class="na">ccs</span><span class="err">.</span><span class="na">neu</span><span class="err">.</span><span class="na">edu</span><span class="err">"</span><span class="p">&gt;</span>types@"@"ccs.neu.edu<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"http://www.ccs.neu.edu/home/types"</span><span class="p">&gt;</span>www.ccs.neu.edu/home/types<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-muted col-md-offset-5"</span><span class="p">&gt;</span> + Joined 2014 + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-bio"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>I like constructions .... <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new definition uses a helper function with keyword arguments for each &ldquo;field&rdquo; of the person:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@person</span><span class="p">[</span><span class="kd">#:name</span> <span class="s2">"Ben Greenman"</span> + <span class="kd">#:title</span> <span class="s2">"Advisor: Matthias Felleisen"</span> + <span class="kd">#:e-mail</span> <span class="s2">"types@ccs.neu.edu"</span> + <span class="kd">#:website</span> <span class="s2">"http://ccs.neu.edu/home/types"</span> + <span class="kd">#:history</span> <span class="n">@list</span><span class="p">[</span><span class="s2">"Joined 2014"</span><span class="p">]</span> + <span class="kd">#:img</span> <span class="s2">"ben_greenman.jpg"</span><span class="p">]{</span> + <span class="n">I</span> <span class="n">like</span> <span class="n">constructions</span> <span class="n">....</span> +<span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="3-less-string-formatting">3. Less String-Formatting</h4> + +<p>Before, the code did a lot of string formatting (<a href="https://github.com/nuprl/website/commit/a0600d#diff-1921e33ce89be28dd277cf1c7880d1beL9">link</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/creatingunits.html#(form._((lib._racket/unit..rkt)._link))" style="color: inherit">link</a></span> <span class="n">url</span> <span class="n">body</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-append))" style="color: inherit">string-append</a></span> <span class="s2">"&lt;a href=</span><span class="se">\"</span><span class="s2">"</span> <span class="n">url</span> <span class="s2">"</span><span class="se">\"</span><span class="s2">&gt;"</span> <span class="n">body</span> <span class="s2">"&lt;/a&gt;"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new code has no need for such helper functions.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@a</span><span class="p">[</span><span class="n">href:</span> <span class="n">url</span> <span class="n">body</span><span class="p">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="bottom-line">Bottom Line</h4> + +<p>Scribble is a good language for making static HTML pages.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://docs.racket-lang.org/pollen/index.html">Pollen</a></li> + <li><a href="https://github.com/vishesh/racketscript">RacketScript</a></li> + <li>Other websites built using <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a>: (1) <a href="http://nanopass.org/">nanopass.github.io</a> (<a href="https://github.com/nanopass/nanopass.github.io">source code</a>), (2) <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> (<a href="https://github.com/nuprl/gtp">source code</a>).</li> + <li><a href="http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/">Notes from a Gradual Typing Across the Spectrum PI meeting</a></li></ul> + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + + urn:http-prl-ccs-neu-edu:-blog-2016-11-30-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages + 2016-11-30T15:24:45Z + 2016-11-30T15:24:45Z + + Jonathan Schuster + + + CompCert Overview + + urn:http-prl-ccs-neu-edu:-blog-2016-10-11-compcert-overview + 2016-10-11T17:41:16Z + 2016-10-11T17:41:16Z + + Ben Greenman + +<p>If you are interested in learning about the <em>internals</em> of the CompCert C compiler but would rather not read its source code, this post is for you.</p> +<!-- more--> + +<p>(This is a public service announcement.)</p> + +<p>Last fall, I gave a short lecture on the 2006 paper <a href="http://gallium.inria.fr/~xleroy/publi/compiler-certif.pdf">&ldquo;Formal Certification of a Compiler Back-End&rdquo;</a> by Xavier Leroy for Amal Ahmed&rsquo;s <a href="http://www.ccs.neu.edu/home/amal/course/7480-f15/">&ldquo;Special Topics in Programming Languages&rdquo;</a> class. Rather than present CompCert as it existed in 2006, I read the documentation and source code for <a href="https://github.com/AbsInt/CompCert/releases/tag/v2.5">CompCert 2.5</a> (released June 2015). The lecture then focused on three questions:</p> + +<ul> + <li>What subset of C does CompCert handle, today?</li> + <li>What optimizing passes does CompCert perform?</li> + <li>What is the &ldquo;correctness theorem&rdquo; for CompCert, and what does this theorem mean?</li></ul> + +<p>My notes for the lecture give a &ldquo;mid-level&rdquo; summary of the compiler &mdash; there are more details than you&rsquo;ll find in papers, but it&rsquo;s (hopefully!) easier to read than the source code. The document is also hyperlinked to locations in the <a href="https://github.com/AbsInt/CompCert">CompCert GitHub repository</a>.</p> + +<p>Here is the document:</p> + +<blockquote> + <p> <a href="http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf">http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf</a></p></blockquote> + +<p>And here is a table-of-contents:</p> + +<ol> + <li>Motivation, details of the source and target languages, high-level guarantees</li> + <li>Compiler pipeline, optimizing passes, links intermediate language grammars and Coq theorems</li> + <li>Background on compiler correctness</li> + <li>CompCert&rsquo;s correctness, properties that CompCert does <strong>not</strong> guarantee</li> + <li>Recent (2006 &ndash; 2015) work in the CompCert ecosystem</li></ol> + +<p>The document ends with a short description of two other research projects that have grown into &ldquo;industry software&rdquo; and a link to Xaver Leroy&rsquo;s <a href="https://www.cs.uoregon.edu/research/summerschool/summer12/curriculum.html">OPLSS lectures on certified compilers</a>. Enjoy!</p> + + Tutorial: Zero to Sixty in Racket + + urn:http-prl-ccs-neu-edu:-blog-2016-08-02-tutorial-zero-to-sixty-in-racket + 2016-08-02T01:29:11Z + 2016-08-02T01:29:11Z + + Ben Greenman + +<p>Racket is excellent for incrementally growing scripts into full-fledged programs. +This post steps through the evolution of one small program and highlights the + Racket tools that enable incremental advances.</p> +<!--more--> + +<p></p> + +<div class="SIntrapara">Why should anyone use <a href="http://racket-lang.org/">Racket</a>? +There are two reasons: +</div> + +<div class="SIntrapara"> + <ol> + <li> + <p>You have a problem that can only be solved with Racket&rsquo;s language-building tools</p></li> + <li> + <p>Racket is a nice language to program in. +(Has lexical scope, parentheses, <a href="http://con.racket-lang.org">active users</a>...)</p></li></ol></div> + +<p>My favorite part of Racket is how it supports a certain development style of + evolving scripts into programs. +When I start coding (after design, before debugging), I can focus the problem at hand. +Next come examples, unit tests, and types to be sure the solution is correct. +Finally, I worry about aesthetics, efficiency, and how the solution can be used as a library in a larger context.</p> + +<p>Bottom line: with Racket, my coding is aligned with my priorities. +And as I transition from "no code" to "working code" to "robust code" to "re-usable code", + the program is almost always runnable.</p> + +<h1><a name="(part._.Problem__.A_.K.W.I.C_.Index_.Production_.System)"></a>Problem: A KWIC Index Production System</h1> + +<p>A KWIC index system +reads input from a file, +divides each line of the file into whitespace-separated words, +and outputs (in alphabetical order) all circular shifts of all lines.</p> + +<p>The first circular shift of a line <span class="RktWrap"><span class="RktVal">"A B C"</span></span> is the line <span class="RktWrap"><span class="RktVal">"B C A"</span></span>. +The second circular shift is <span class="RktWrap"><span class="RktVal">"C A B"</span></span>.</p> + +<p>Building a KWIC index is a historical problem. +According to <a href="https://www.cs.umd.edu/class/spring2003/cmsc838p/Design/criteria.pdf">D.L. Parnas (1972)</a>:</p> + +<blockquote class="SubFlow"> + <p>Except under extreme circumstances (huge data base, no supporting software) +such a system could be implemented by a good programmer within a week or two.</p></blockquote> + +<p>See also: <a href="https://yanniss.github.io/law.html">Yannis&rsquo;s Law</a>.</p> + +<p>Today, I bet only <a href="http://wiki.portal.chalmers.se/agda/pmwiki.php">Agda</a> and <a href="https://scratch.mit.edu/">Scratch</a> + programmers would need the full two weeks. +We&rsquo;ll be done in 20 minutes.</p> + +<h1><a name="(part._.A_.Script)"></a>A Script</h1> + +<p>To start, open a file and type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>You can name the file anything, like <span class="stt">kwic.rkt</span> or <span class="stt">rkt.kwic</span> or <span class="stt">foo</span>. +Racket doesn&rsquo;t care, + but it does need the <span class="stt">#lang</span> line to read the contents of the file.</p> + +<p>Though, you should use the <span class="stt">.rkt</span> extension.</p> + +<p>The first part of the solution is a function to read input from a file into a + list of strings for further processing. +The built-in function <a href="http://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28lib._racket%2Ffile..rkt%29._file-~3elines%29%29">file-&gt;lines</a> + does exactly this, but we&rsquo;ll for-loop instead.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="hspace">&nbsp;</span><span class="RktSym">filename</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="hspace">&nbsp;</span><span class="RktSym">filename</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>When called with a filename like <span class="RktWrap"><span class="RktVal">"heart-of-darkness.txt"</span></span>, the function + uses <a href="http://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a> to build a list of lines by reading from a port + with <a href="http://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a>. +The port is the data from <span class="RktWrap"><span class="RktSym">filename</span></span>, thanks to <a href="http://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a>.</p> + +<p>Next is a function to convert a list of strings into a list of lists of words. +Here we&rsquo;ll just use library functions.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktSym">lines</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="hspace">&nbsp;</span><span class="RktSym">string-split</span><span class="hspace">&nbsp;</span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>By default, <a href="http://docs.racket-lang.org/reference/strings.html#%28def._%28%28lib._racket%2Fstring..rkt%29._string-split%29%29">string-split</a> divides a string into a list of whitespace-separated substrings. +You can always supply a different delimiter, or use <a href="http://docs.racket-lang.org/reference/regexp.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._regexp-split%29%29">regexp-split</a> + to divide by a regular expression.</p> + +<p>Two tasks left! +First we generate all circular shifts for a list of strings <span class="RktWrap"><span class="RktSym">words</span></span> + by folding up a list with one shift of <span class="RktWrap"><span class="RktSym">words</span></span> for each word.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Second, we alphabetize and print the shifts.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">destruct multiple values</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">first list empty, don</span><span class="RktCmt">'</span><span class="RktCmt">t care about second</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#t</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">first list non-empty, second empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1-rest</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="hspace">&nbsp;</span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktVal">" "</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">for-each is like map, but returns (void)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="hspace">&nbsp;</span><span class="RktSym">display-words</span><span class="hspace">&nbsp;</span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Gluing it all together, here&rsquo;s the full script (with type annotations in comments).</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">type</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">=</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">type</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">=</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Path-String</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Move</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">element</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">last</span><span class="hspace">&nbsp;</span><span class="RktCmt">position</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lexicographic</span><span class="hspace">&nbsp;</span><span class="RktCmt">order</span><span class="hspace">&nbsp;</span><span class="RktCmt">on</span><span class="hspace">&nbsp;</span><span class="RktCmt">equal-length</span><span class="hspace">&nbsp;</span><span class="RktCmt">lists</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Boolean</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">don't</span><span class="hspace">&nbsp;</span><span class="RktCmt">care</span><span class="hspace">&nbsp;</span><span class="RktCmt">about</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">non-empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/booleans.html#%28def._%28%28quote._~23~25kernel%29._not%29%29">not</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._null~3f%29%29">null?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"</span><span class="hspace">&nbsp;</span><span class="RktVal">"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Path-String</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">append*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">End-to-end</span><span class="hspace">&nbsp;</span><span class="RktCmt">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">run-test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"test.txt"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Make</span><span class="hspace">&nbsp;</span><span class="RktCmt">a</span><span class="hspace">&nbsp;</span><span class="RktCmt">file</span><span class="hspace">&nbsp;</span><span class="RktCmt">and</span><span class="hspace">&nbsp;</span><span class="RktCmt">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/when_unless.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._unless%29%29">unless</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._file-exists~3f%29%29">file-exists?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">run-test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Running the file should print:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<h1><a name="(part._.Testing_and_.Submodules)"></a>Testing and Submodules</h1> + +<p>Any top-level expressions in a file can work as unit tests. +The <a href="http://docs.racket-lang.org/reference/booleans.html%3F#%28def._%28%28quote._~23~25kernel%29._equal~3f%29%29">equal?</a> statement below checks whether the first circular shift + of <span class="RktWrap"><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="stt"> </span><span class="RktVal">"B"</span><span class="stt"> </span><span class="RktVal">"C"</span><span class="RktVal">)</span></span> is <span class="RktWrap"><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="stt"> </span><span class="RktVal">"C"</span><span class="stt"> </span><span class="RktVal">"A"</span><span class="RktVal">)</span></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Equality.html#%28def._%28%28quote._~23~25kernel%29._equal~3f%29%29">equal?</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Running the file now prints <span class="RktWrap"><span class="RktVal">#t</span></span> to the console, meaning the test passed. +We can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/exns.html#%28def._%28%28quote._~23~25kernel%29._error%29%29">error</a></span></span> or <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/exns.html#%28def._%28%28quote._~23~25kernel%29._raise-user-error%29%29">raise-user-error</a></span></span> to make failures easier + to notice. +Or we can use the <a href="http://docs.racket-lang.org/rackunit/api.html">RackUnit</a> testing library.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">import the testing library</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>These tests run each time the module does. +If you prefer to run tests only in a specific context, and not when the + module is run or imported as a library, you can move them to a separate + file or into a <a href="http://docs.racket-lang.org/reference/eval-model.html#%28tech._submodule%29">submodule</a>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Open a submodule named </span><span class="RktCmt">'</span><span class="RktCmt">test</span><span class="RktCmt">'</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Running the module normally via <span class="stt">racket kwic.rkt</span> will not run code + in the submodule. +Instead, use <span class="stt">raco test</span> to run the tests.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">raco</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">raco</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._submod%29%29">submod</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"kwic.rkt"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">passed</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The reason we used <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span></span>, instead of Racket&rsquo;s <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%29%29">module</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%2A%29%29">module*</a></span></span> + forms is that <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span></span> inherits the language and namespace of its + containing module and can be incrementally extended. +This way, we can keep tests near the relevant code.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"is"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"as"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"does"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"as"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"does"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"is"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p><a href="http://docs.racket-lang.org/rackunit/api.html">RackUnit</a> in a + separate file or <span class="RktWrap"><span class="RktSym">test</span></span> submodule is the unofficial standard for testing + Racket programs.</p> + +<h1><a name="(part._.Recognizing_.Patterns__.Avoiding_.Repetition)"></a>Recognizing Patterns, Avoiding Repetition</h1> + +<p>Every unit test we&rsquo;ve written uses <a href="http://docs.racket-lang.org/rackunit/api.html#%28def._%28%28lib._rackunit%2Fmain..rkt%29._check-equal~3f%29%29">check-equal?</a>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"hello"</span><span class="hspace">&nbsp;</span><span class="RktVal">"world"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">" lost "</span><span class="hspace">&nbsp;</span><span class="RktVal">" in "</span><span class="hspace">&nbsp;</span><span class="RktVal">"space"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"lost"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"in"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"space"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"something"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>These tests follow a simple pattern that we can express as a <span class="emph">syntax rule</span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._define-syntax-rule%29%29">define-syntax-rule</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="hspace">&nbsp;</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="RktPn">]</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/begin.html#%28form._%28%28quote._~23~25kernel%29._begin%29%29">begin</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"hello"</span><span class="hspace">&nbsp;</span><span class="RktVal">"world"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">" out "</span><span class="hspace">&nbsp;</span><span class="RktVal">" in "</span><span class="hspace">&nbsp;</span><span class="RktVal">"the ozone"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"out"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"in"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"the"</span><span class="hspace">&nbsp;</span><span class="RktVal">"ozone"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"something"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span></span> are not pseudocode! +They denote Kleene-star repetition, like a sextile (<span class="RktWrap"><span class="RktVal">"*"</span></span>) in a regular expression. +In this case, the input pattern is a sequence of lists with two S-expressions, <span class="RktWrap"><span class="RktSym">i</span></span> and + <span class="RktWrap"><span class="RktSym">o</span></span>. +Uses of <span class="RktWrap"><span class="RktSym">i</span></span> and <span class="RktWrap"><span class="RktSym">o</span></span> in the rule must be followed by one <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span></span> to splice + the captured S-expressions into the result.</p> + +<p>Many languages offer higher-order functions and polymorphism to abstract common + behaviors. +Syntax extensions are a different way to avoid repeating yourself. +After 30 years, we are still discovering what syntax extensions are useful for.</p> + +<p>See this <a href="https://groups.google.com/forum/#!topic/racket-users/ss20lwfUhjs/discussion">recent Racket mailing list post</a> for some applications.</p> + +<h1><a name="(part._.Adding_.Static_.Types)"></a>Adding Static Types</h1> + +<p>Changing the <a href="http://docs.racket-lang.org/reference/reader.html#%28idx._%28gentag._79._%28lib._scribblings%2Freference%2Freference..scrbl%29%29%29"><span class="stt">#lang</span></a> line to <span class="RktWrap"><span class="RktSym">typed/racket</span></span> adds static type-checking to our program. +If we only change the language and run the code as-is, there will be type errors. +But we can use submodules again to incrementally check our design with types.</p> + +<p>Note: <a href="http://docs.racket-lang.org/ts-reference/Typed_Regions.html">typed regions</a> + are another way to embed typed code into untyped contexts.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%29%29">module</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">typed/racket</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Need</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">annotate:</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-</span><span class="hspace">&nbsp;</span><span class="RktCmt">function</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameters</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-</span><span class="hspace">&nbsp;</span><span class="RktCmt">for-loop</span><span class="hspace">&nbsp;</span><span class="RktCmt">return</span><span class="hspace">&nbsp;</span><span class="RktCmt">types</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Next</span><span class="hspace">&nbsp;</span><span class="RktCmt">migration</span><span class="hspace">&nbsp;</span><span class="RktCmt">step:</span><span class="hspace">&nbsp;</span><span class="RktCmt">move</span><span class="hspace">&nbsp;</span><span class="RktCmt">other</span><span class="hspace">&nbsp;</span><span class="RktCmt">untyped</span><span class="hspace">&nbsp;</span><span class="RktCmt">functions</span><span class="hspace">&nbsp;</span><span class="RktCmt">here</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._all-defined-out%29%29">all-defined-out</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">t</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">&lt;rest</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">file</span><span class="hspace">&nbsp;</span><span class="RktCmt">omitted&gt;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>After scooping all functions into the Typed Racket bubble, we can remove the + submodule declaration and change <span class="stt">#lang racket</span> to <span class="stt">#lang typed/racket</span>.</p> + +<h1><a name="(part._.Finally__a_.Library)"></a>Finally, a Library</h1> + +<p>Other modules can import our functions if we use a <a href="http://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a> statement. +By <a href="https://docs.racket-lang.org/style/Units_of_Code.html">convention</a>, exports belong at the top of a file.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">#lang</span><span class="hspace">&nbsp;</span><span class="RktMeta">typed/racket</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-index</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">&lt;definitions</span><span class="hspace">&nbsp;</span><span class="RktCmt">here&gt;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Then any typed or untyped module can use <span class="RktWrap"><span class="RktSym">kwic-index</span></span> by writing + <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="stt"> </span><span class="RktVal">"kwic.rkt"</span><span class="RktPn">)</span></span>.</p> + +<p>As a finishing touch, we can use the <a href="http://docs.racket-lang.org/reference/Command-Line_Parsing.html">racket/cmdline</a> library + inside a <span class="stt">main</span> submodule to give a basic front-end interface. +Similar to <span class="stt">module+ test</span>, a <span class="stt">module+ main</span> declares code that + inherits the file&rsquo;s bindings and language but is only run when the program + is executaed.</p> + +<p>Here is the complete typed and tested code listing. +The <span class="RktWrap"><span class="RktSym">main</span></span> submodule is at the bottom.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/ts-reference/index.html"><span class="RktSym">typed/racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">typed/rackunit</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._define-syntax-rule%29%29">define-syntax-rule</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">o</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/begin.html#%28form._%28%28quote._~23~25kernel%29._begin%29%29">begin</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">o</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">define-type</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">define-type</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">make-temporary-file</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"The</span><span class="hspace">&nbsp;</span><span class="RktVal">Nellie,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"a</span><span class="hspace">&nbsp;</span><span class="RktVal">cruising</span><span class="hspace">&nbsp;</span><span class="RktVal">yawl,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"swung</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">her</span><span class="hspace">&nbsp;</span><span class="RktVal">anchor</span><span class="hspace">&nbsp;</span><span class="RktVal">without</span><span class="hspace">&nbsp;</span><span class="RktVal">a</span><span class="hspace">&nbsp;</span><span class="RktVal">flutter</span><span class="hspace">&nbsp;</span><span class="RktVal">of</span><span class="hspace">&nbsp;</span><span class="RktVal">sails,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"and</span><span class="hspace">&nbsp;</span><span class="RktVal">was</span><span class="hspace">&nbsp;</span><span class="RktVal">at</span><span class="hspace">&nbsp;</span><span class="RktVal">rest."</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">actual</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expect</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">file-&gt;lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._delete-file%29%29">delete-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">actual</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expect</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#{</span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta">::</span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"hello"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"world"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Move</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">element</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">last</span><span class="hspace">&nbsp;</span><span class="RktCmt">position</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lexicographic</span><span class="hspace">&nbsp;</span><span class="RktCmt">order</span><span class="hspace">&nbsp;</span><span class="RktCmt">on</span><span class="hspace">&nbsp;</span><span class="RktCmt">equal-length</span><span class="hspace">&nbsp;</span><span class="RktCmt">lists</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Boolean</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">don't</span><span class="hspace">&nbsp;</span><span class="RktCmt">care</span><span class="hspace">&nbsp;</span><span class="RktCmt">about</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">non-empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"</span><span class="hspace">&nbsp;</span><span class="RktVal">"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._open-output-string%29%29">open-output-string</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._get-output-string%29%29">get-output-string</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A\nB</span><span class="hspace">&nbsp;</span><span class="RktVal">C\n"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"D"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"D"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">append*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._open-output-string%29%29">open-output-string</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">make-temporary-file</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._delete-file%29%29">delete-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._get-output-string%29%29">get-output-string</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">string-join</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"if</span><span class="hspace">&nbsp;</span><span class="RktVal">this</span><span class="hspace">&nbsp;</span><span class="RktVal">imagine"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"this</span><span class="hspace">&nbsp;</span><span class="RktVal">imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to\n"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"\n"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">main</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket/cmdline</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Parameterof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Any</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28def._%28%28quote._~23~25kernel%29._make-parameter%29%29">make-parameter</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">command-line</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:program</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"kwic</span><span class="hspace">&nbsp;</span><span class="RktVal">index"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:once-each</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktVal">"-o"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"--output"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">user-supplied</span><span class="hspace">&nbsp;</span><span class="RktCmt">input</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"Write</span><span class="hspace">&nbsp;</span><span class="RktVal">output</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">file"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">update</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameter</span><span class="hspace">&nbsp;</span><span class="RktCmt">*output-to*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:args</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">*output-to*</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">read</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameter</span><span class="hspace">&nbsp;</span><span class="RktCmt">*output-to*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3f%29%29">string?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._open-output-file%29%29">open-output-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">cast</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/when_unless.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._when%29%29">when</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3f%29%29">string?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._close-output-port%29%29">close-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">Sample interactions: +</div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">kwic</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expects</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">&lt;file-name&gt;</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">on</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">the</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">command</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quasiquote.html#%28form._%28%28quote._~23~25kernel%29._unquote%29%29">,</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">given</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">arguments</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">echo</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"It</span><span class="hspace">&nbsp;</span><span class="RktVal">is</span><span class="hspace">&nbsp;</span><span class="RktVal">a</span><span class="hspace">&nbsp;</span><span class="RktVal">truth</span><span class="hspace">&nbsp;</span><span class="RktVal">universally</span><span class="hspace">&nbsp;</span><span class="RktVal">acknowledged"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">pride-and-prejudice.txt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-o</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">pride-and-prejudice.txt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">wc</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-l</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktVal">6</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span></td></tr></tbody></table></div></div> + +<h1><a name="(part._.Closing)"></a>Closing</h1> + +<p>We started with functions, wrote (and quarantined) unit tests, + reinforced our design with types, and added a command-line interface. +Going forward we could add <a href="http://docs.racket-lang.org/scribble/index.html">Scribble</a> documentation and share our work as a <a href="http://pkgn.racket-lang.org/">package</a>.</p> + +<p></p> + +<div class="SIntrapara">For more on building languages with Racket: +</div> + +<div class="SIntrapara"> + <ul> + <li> + <p><a href="http://www.hashcollision.org/brainfudge/">Fudging up a Racket (html)</a></p></li> + <li> + <p><a href="http://dl.acm.org/authorize?6529547">Creating Languages in Racket (pdf)</a></p></li> + <li> + <p><a href="http://www.ccs.neu.edu/home/matthias/manifesto/">The Racket Manifesto (html)</a></p></li> + <li> + <p><a href="http://www.terohasu.net/hasu-flatt--els16--preprint.pdf">Source-to-Source Compilation via Submodules (pdf)</a></p></li></ul></div> + + Tutorial: Racket FFI, part 3 + + urn:http-prl-ccs-neu-edu:-blog-2016-07-11-tutorial-racket-ffi-part-3 + 2016-07-11T17:33:40Z + 2016-07-11T17:33:40Z + + Asumu Takikawa + +<p>This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a> +and part 2 +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a>.</p> + +<p>In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.</p> +<!--more--> + +<p>As in the second post, let&rsquo;s start with some prologue code that establishes +the definitions from the previous two posts. But first, I&rsquo;m getting tired of +writing the <span class="stt">#:c-id identifier</span> notation for the underscored C function +names.</p> + +<p>Instead, let&rsquo;s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">$</span><span class="hspace">&nbsp;</span><span class="RktMeta">raco</span><span class="hspace">&nbsp;</span><span class="RktMeta">pkg</span><span class="hspace">&nbsp;</span><span class="RktMeta">install</span><span class="hspace">&nbsp;</span><span class="RktMeta">ffi-definer-convention</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>or you can just execute the following snippet in Racket:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pkg</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">pkg-install-command</span><span class="hspace">&nbsp;</span><span class="RktPn">#:skip-installed</span><span class="hspace">&nbsp;</span><span class="RktVal">#t</span><span class="hspace">&nbsp;</span><span class="RktVal">"ffi-definer-convention"</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Downloading repository git://github.com/takikawa/racket-ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: version: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: platform: x86_64-linux [3m]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: target machine: racket</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: installation name: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: variants: 3m</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main collects: /usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: collects paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main pkgs: /usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: pkgs paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: links files: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main docs: /usr/share/racket/doc</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- updating info-domain tables ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- pre-installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing foreign libraries --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing shared files ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- compiling collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- parallel build using 4 jobs ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 making: &lt;pkgs&gt;/ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- creating launchers --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing man pages --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- building documentation --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 running: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 rendering: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 2 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/local-redirect.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 1 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/release.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/search.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/start.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- post-installing collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This will install the package and compile its contents. If you&rsquo;re curious, the +docs for the package are available +<a href="http://docs.racket-lang.org/ffi-definer-convention/index.html">here</a>.</p> + +<p><span style="font-weight: bold">Note:</span> if you&rsquo;ve never installed a package before, you may want to glance +at the +<a href="http://docs.racket-lang.org/pkg/getting-started.html">package system docs</a>. +A tl;dr of packages is that they bundle Racket <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/collects.html#%28tech._collection%29"><span class="techinside">collections</span></a>, which are +sets of modules that you can refer to in a location-independent fashion such as +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> or <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28mod-path._racket%2Flist%29"><span class="RktSym">racket/list</span></a></span>.</p> + +<p>Anyhow, here&rsquo;s the prologue code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">avoid conflict with below</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._except-in%29%29">except-in</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">define-ffi-definer</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the new 3rd-party pkg</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi-definer-convention</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">describes how to transform from</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Racket to C ids</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:make-c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">convention:hyphen-&gt;underscore</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">note lack of #:c-id keyword arguments</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">(_cairo_t -&gt; Void) -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">do some drawing and give us the pict</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">f</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">f</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Notice that the <span class="RktWrap"><span class="RktSym">define-cairo</span></span> forms don&rsquo;t have any <span class="stt">#:c-id</span> keywords +anymore. Instead, the prologue code uses an overriden <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> +from my package that supports a <span class="stt">#:make-c-id</span> keyword that lets you specify +a naming convention to follow.</p> + +<p>Also, instead of creating a single bitmap and drawing into it, we now have a +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> function that takes a drawing function. When called, +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> will call the given function with a new bitmap object and +return the result.</p> + +<p>Now let&rsquo;s get to the main point of this blog post. Let&rsquo;s say that we want to play +with Cairo <a href="https://www.cairographics.org/manual/cairo-Paths.html">path</a> +objects this time. A path is +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-t">defined</a> +as a struct with the following structure:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_status_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">status</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it&rsquo;s +useful to define C types for the types of values in the path struct&rsquo;s fields. First, +let&rsquo;s specify that a <span class="stt">cairo_status_t</span> is an integer type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>It&rsquo;s actually an enum, but for the examples in this post we don&rsquo;t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-data-t">path data objects</a>. +Each path data object is a <span class="stt">cairo_path_data_t</span>, +which is specified with a C union:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">union</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">_cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_type_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">type</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">length</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">header</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">point</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Helpfully, the FFI library comes with support for unions with the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span></span> type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It&rsquo;s pretty +straightforward to specify this type too:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the path data type is just an enum</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">move-to</span><span class="hspace">&nbsp;</span><span class="RktVal">line-to</span><span class="hspace">&nbsp;</span><span class="RktVal">curve-to</span><span class="hspace">&nbsp;</span><span class="RktVal">close-path</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the header case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the point case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>There&rsquo;s a new type constructor here so let me explain that first. +The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span></span> constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span>, this constructor doesn&rsquo;t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.</p> + +<p>Each of the path data structs in the path data array will be manipulated with +the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">create a union from a list of doubles</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1.3</span><span class="hspace">&nbsp;</span><span class="RktVal">5.8</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">source type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">target type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;union&gt;</span></p></td></tr></tbody></table></div> + +<p>This snippet first construct a union object (via the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> +type) using a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span></span>. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it&rsquo;s an easy way +to generate a union object.</p> + +<p>The second line shows that a union prints as an opaque object. You can&rsquo;t do +anything with a union in Racket unless you project it to one of the sub-cases with +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></span> function. +This projection is <span class="emph">unsafe</span>, in the sense that if you don&rsquo;t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.</p> + +<p>More concretely, let&rsquo;s see what happens if we try to extract a value out of the +union both correctly and incorrectly:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">correct (matches construction)</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">cases are zero-indexed and ordered as written</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(1.3 5.8)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">incorrect, error</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">enum:int-&gt;_cairo_path_data_type_t: expected a known</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">#&lt;ctype:ufixint&gt;, got: 3435973837</span></p></td></tr></tbody></table></div> + +<p>Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.</p> + +<p>With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.</p> + +<p>With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it&rsquo;s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.</p> + +<p>So before moving on, let&rsquo;s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we&rsquo;ll figure out how to +deal with the array itself.</p> + +<h1><a name="(part._.Some_low-level_operations)"></a>Some low-level operations</h1> + +<p>Since we still don&rsquo;t have a C type for <span class="stt">cairo_path_t</span>, let&rsquo;s go ahead +and make a simple one where we punt on the work of specifying the array +type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In this type, we have specified the array as a bare <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span>. +For some added safety, we could also use something like +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__cpointer%29%29">_cpointer</a></span><span class="stt"> </span><span class="RktVal">'</span><span class="RktVal">cairo_status_t</span><span class="RktPn">)</span></span>, which sets up a tagged pointer +type like we saw in the first blog post with +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>.</p> + +<p>We&rsquo;ve seen the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type before, but haven&rsquo;t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.</p> + +<p>Before we get to that, let&rsquo;s go ahead and set up an FFI binding for +<span class="stt">cairo_copy_path</span> so that we can obtain a path struct to manipulate:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Do stuff to make the current</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">path non-empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Get the current path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/set_.html#%28form._%28%28quote._~23~25kernel%29._set%21%29%29">set!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Stroke clears the path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">so do it last</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<p>Note that <span class="RktWrap"><span class="RktSym">cairo-copy-path</span></span> gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span>, which +lets you dereference a pointer and access it at some concrete C type.</p> + +<p><span style="font-weight: bold">Note:</span> the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> function also takes an optional +offset argument which we will be used in an example later.</p> + +<p>For example, we can use <span class="RktWrap"><span class="RktSym">a-path</span></span> as a <span class="RktWrap"><span class="RktSym">_simple_cairo_path_t</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(0 #&lt;cpointer&gt; 8)</span></p></td></tr></tbody></table></div> + +<p>And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> again with an array type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">array</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">length field</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">array</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(#&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt;)</span></p></td></tr></tbody></table></div> + +<p>The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(move-to 2)</span></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(50.0 50.0)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">nonsense data here, wrong union case</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(4.2439915824246e-314 0.0)</span></p></td></tr></tbody></table></div> + +<p>One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.</p> + +<p>An alternative is to define a <span class="emph">custom C type</span> that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don&rsquo;t need to think about applying helper functions and +dereferencing pointers.</p> + +<h1><a name="(part._.Custom_.C_types)"></a>Custom C types</h1> + +<p>I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.</p> + +<p>In this way, it&rsquo;s possible to conduct interesting conversions, such as +dereferencing union objects automatically.</p> + +<p>Now let&rsquo;s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.</p> + +<p>First, we&rsquo;ll start by defining a struct type for the Racket representation +of Cairo paths:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define-struct.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._struct%29%29">struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ptr</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:property</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._prop~3asequence%29%29">prop:sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The representation will store one field <span class="RktWrap"><span class="RktSym">ptr</span></span> which, as the name +suggests, will store a pointer value. We&rsquo;ll see what to do with this +pointer later.</p> + +<p>This definition uses a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/structprops.html#%28tech._structure._type._property%29"><span class="techinside">structure type property</span></a> to make instances of +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> automatically work as sequences. This means that you +can iterate over them with a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span></span> loop or apply <span class="RktWrap"><span class="RktSym">sequence-ref</span></span> +on them. The property takes a function that takes an instance of the struct +type itself (here <span class="RktWrap"><span class="RktSym">p</span></span>) and that returns a sequence.</p> + +<p>We&rsquo;ll later define the <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span> function that will actually +construct the relevant sequence for us. For now, let&rsquo;s see how to construct +the C type given this struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract pointer out of representation</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">racket-&gt;c</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Just apply the Racket constructor</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">c-&gt;racket</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">racket-&gt;c</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">c-&gt;racket</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The base type for this <span class="RktWrap"><span class="RktSym">_cairo_path_t</span></span> is a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type. Since +the Cairo API returns pointers to new path values, it&rsquo;s hard to avoid using some +kind of pointer type as the base type here.</p> + +<p>This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we&rsquo;ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.</p> + +<p>The real work is done by the helper function that makes a +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> instance work as a sequence.</p> + +<p>Starting top-down, let&rsquo;s look at the definition of <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Cairo-Path -&gt; Sequence</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match-define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/values.html#%28def._%28%28quote._~23~25kernel%29._values%29%29">values</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The first thing the function does is extract the pointer out of the +representation, and then immediately calls <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.</p> + +<p><span style="font-weight: bold">Note:</span> in case you&rsquo;re not very familiar with Racket pattern matching, the +<span class="RktWrap"><span class="RktSym">match-define</span></span> form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml&rsquo;s <span class="stt">let</span> statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +<a href="http://docs.racket-lang.org/reference/match.html#%28form._%28%28lib._racket%2Fmatch..rkt%29._match-define%29%29">docs</a>.</p> + +<p>After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> function. Essentially, <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.</p> + +<p><span style="font-weight: bold">Note:</span> technically <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it&rsquo;s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.</p> + +<p>In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the <span class="RktWrap"><span class="RktSym">pos-&gt;element</span></span> function +and its helpers:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer -&gt; Integer -&gt; Element</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract the data path header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Length includes header, so subtract 1</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._sub1%29%29">sub1</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._add1%29%29">add1</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer Integer Integer -&gt; (Listof Data)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">offset argument</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in <span class="RktWrap"><span class="RktSym">get-points</span></span> +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.</p> + +<p>Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Note that determining the end of the sequence is very easy. It&rsquo;s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="stt"> </span><span class="RktSym">pos</span><span class="stt"> </span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></span>.</p> + +<p>Now we can try using a path as a sequence:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Using path as a sequence</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">elem</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="hspace">&nbsp;</span><span class="RktSym">elem</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 50.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (206.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (115.0 115.0))</span></p></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span></span> type.</p> + +<p>That concludes part 3 of the FFI tutorial. Hopefully you&rsquo;re now equipped +to deal with union types and custom C types. If not, see the +<a href="http://docs.racket-lang.org/foreign/index.html">FFI reference</a> +for more details on +<a href="http://docs.racket-lang.org/foreign/C_Union_Types.html">unions</a> +and +<a href="http://docs.racket-lang.org/foreign/ctype.html">custom C types</a>.</p> + +<p><span class="emph">Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!</span></p> + + Tutorial: Racket FFI, Part 2 + + urn:http-prl-ccs-neu-edu:-blog-2016-06-29-tutorial-racket-ffi-part-2 + 2016-06-29T18:48:17Z + 2016-06-29T18:48:17Z + + Asumu Takikawa + +<p>This is part 2 of my tutorial on using the Racket FFI. If you haven&rsquo;t read +part 1 yet, you can find it +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a>. +<span style="font-weight: bold">Update:</span> part 3 is also now available +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.</p> +<!--more--> + +<p>First, here&rsquo;s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">bitmap magic</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_create</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Bitmap -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a helper for displaying the bitmap</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<h1><a name="(part._.Dashes_and_array_arguments)"></a>Dashes and array arguments</h1> + +<p>To start off, let&rsquo;s look at another C example from the Cairo +<a href="https://www.cairographics.org/samples/">samples page</a>. +This time we will look at the "dash" example, which has a +use of an input array:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">sizeof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">)</span><span class="RktMeta">/sizeof</span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktVal">0</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">50.0</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">25.6</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_rel_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">102.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_curve_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The most interesting function here is <span class="stt">cairo_set_dash</span>, which takes an +array argument. The only other new functions are <span class="stt">cairo_rel_line_to</span> +and <span class="stt">cairo_curve_to</span> which have very straightforward C types:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-rel-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_rel_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-curve-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_curve_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Meanwhile, the C type signature for <span class="stt">cairo_set_dash</span> from the Cairo +docs looks like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Something to note about the arguments is that <span class="stt">num_dashes</span> +encodes the length of the array <span class="stt">dashes</span>. This will come up later when +we want to make the C type for this function more convenient.</p> + +<p>On the Racket side, it&rsquo;s natural to represent the array of dashes as either a +list or <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/vectors.html#%28tech._vector%29"><span class="techinside">vector</span></a> of numbers. Given that, a fairly literal translation of +the C type above might look like the following:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This type includes a type constructor we haven&rsquo;t seen yet: <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span>. +This is a so-called <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28tech._custom._function._type%29"><span class="techinside">custom function type</span></a> that has special meaning +inside of a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.</p> + +<p>Since we only want to provide a list from the Racket side to C, we&rsquo;ll use +the <span class="RktWrap"><span class="RktSym">i</span></span> input mode. We can then call the function like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">4</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Note that because of how we defined the type of <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it&rsquo;s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.</p> + +<p>Luckily, the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type constructor actually lets you do this with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">name</span><span class="stt"> </span><span class="RktSym">:</span><span class="stt"> </span><span class="RktSym">type</span><span class="RktPn">)</span></span> syntax for naming arguments in combination with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">type</span><span class="stt"> </span><span class="RktVar">=</span><span class="stt"> </span><span class="RktSym">expr</span><span class="RktPn">)</span></span> syntax for supplying computed arguments:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">name this argument for later uses in the type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a computed argument position</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3d%29%29">=</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>When a computed argument is specified with a <span class="RktWrap"><span class="RktVar">=</span></span>, it&rsquo;s not necessary to provide +the argument on the Racket side. So <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> is now an arity 3 +function that can be called like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This means we&rsquo;ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it&rsquo;s also possible to use Racket vectors instead of lists by using +the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__vector%29%29">_vector</a></span></span> type constructor.</p> + +<p>Putting it all together, we can reproduce the dashes example like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">25.6</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-rel-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-1</span>02.4</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-curve-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Result_arguments_and_.C_structs)"></a>Result arguments and C structs</h1> + +<p>For some more advanced FFI hacking, let&rsquo;s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Our challenge is to make a Racket function that takes a string (let&rsquo;s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let&rsquo;s just scale the +text to fit the width and assume the height will be okay.</p> + +<p>To implement the key step of measuring the text size, we can use the +<a href="https://www.cairographics.org/manual/cairo-text.html#cairo-text-extents"><span class="stt">cairo_text_extents</span></a> +function. Its type signature is as follows:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">char</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*utf8,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*extents</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The interesting part of this signature is that +<a href="https://www.cairographics.org/manual/cairo-cairo-scaled-font-t.html#cairo-text-extents-t"><span class="stt">cairo_text_extents_t</span></a> +is a struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">Cairo</span><span class="hspace">&nbsp;</span><span class="RktCmt">docs</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">height</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>We haven&rsquo;t yet seen how to handle C structs with the FFI, but it&rsquo;s not +too tricky. Support for C structs comes built-in and will look familiar +if you&rsquo;re used to Racket structs. We can directly translate the documented +definition above into a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span> declaration:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the leading underscore is mandatory</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">width</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">height</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">x-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">pointer to struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">allows NULL pointer</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer/null</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr></tbody></table></div> + +<p>Along with functions that look like regular Racket struct operations:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a struct constructor</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:make-cairo_text_extents_t&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field selector</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:cairo_text_extents_t-width&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a predicate for the struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t?</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:^TYPE?&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field mutation function</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">set-cairo_text_extents_t-width!</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:set-cairo_text_extents_t-width!&gt;</span></p></td></tr></tbody></table></div> + +<p>With the struct type defined, it&rsquo;s easy to come up with a rudimentary +interface for <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>This style of programming feels awfully imperative though. Since we&rsquo;re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span> FFI wrapper +by combining named arguments, a new <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> type constructor, +and a neat feature of <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> that lets you customize +the return result:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">named args and _ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">ext</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the return result of the C function</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">custom return result for the wrapper</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">ext</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> constructor works like the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span> constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the <span class="RktWrap"><span class="RktSym">o</span></span> mode to <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span>. +In output mode, this type will automatically allocate a new instance of the type +(using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._malloc%29%29">malloc</a></span></span> function) and arrange for it to be passed in as +a pointer.</p> + +<p>The strangest part of this example is that there are now two uses of the +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span></span> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.</p> + +<p>Using this new version of the wrapper is much simpler:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>With that in hand, it&rsquo;s pretty easy to write the function we set out to write:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-show-text</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_show_text</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-scale</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">String -&gt; Void</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">draws a string scaled horizontally</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">20</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">2.0</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._-%29%29"><span class="nobreak">-</span></a></span><span class="hspace">&nbsp;</span><span class="RktVal">256.0</span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym">width</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-scale</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-show-text</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>And to conclude part 2 of this tutorial, here&rsquo;s an example use +of the new <span class="RktWrap"><span class="RktSym">fit-text</span></span> function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktVal">"Saluton, Mondo / Hallo, mundo"</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + + Tutorial: Using Racket's FFI + + urn:http-prl-ccs-neu-edu:-blog-2016-06-27-tutorial-using-racket-s-ffi + 2016-06-27T16:22:11Z + 2016-06-27T16:22:11Z + + Asumu Takikawa + +<p><span style="font-weight: bold">Update:</span> this post is now part of a series. Part 2 is +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a> +and part 3 is +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>I&rsquo;ve seen several people ask for a tutorial on Racket&rsquo;s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +<a href="http://docs.racket-lang.org/foreign/index.html">documentation</a> +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.</p> + +<p>With that in mind, this blog post will provide a step-by-step tutorial +for Racket&rsquo;s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.</p> +<!--more--> + +<p>Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop <a href="http://www.ccs.neu.edu/racket/pubs/scheme04-bo.pdf">paper</a> +that you can read if you&rsquo;re curious about the design.</p> + +<p>The tutorial will focus on using the <a href="https://www.cairographics.org/">Cairo</a> +graphics library, mainly because it comes bundled with Racket.</p> + +<p>To start, let&rsquo;s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo&rsquo;s +<a href="https://www.cairographics.org/samples/">samples page</a>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">30.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">CAIRO_LINE_CAP_ROUND</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This uses the Racket drawing library <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/draw/index.html"><span class="RktSym">racket/draw</span></a></span> to construct +a bitmap object that we&rsquo;ll draw on. The <span class="RktWrap"><span class="RktSym">get-handle</span></span> method just extracts a +low-level Cairo surface value that we can use.</p> + +<p>NB: these code snippets don&rsquo;t come with a <span class="stt">#lang</span> declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy &amp; paste the snippets into your REPL.</p> + +<p>Our first real step is to import the FFI itself:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>As the module name suggests, the FFI is <span class="emph">unsafe</span> and can cause your Racket process +to segfault. If you&rsquo;re following along in DrRacket, you will want to save your file +frequently.</p> + +<p>Next, we can load the Cairo library to obtain a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28tech._foreign._library._value%29"><span class="techinside">foreign-library value</span></a>, which +is a handle that we use to access C values and functions:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Since Cairo has already been loaded by the Racket process because of the +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/gui/index.html"><span class="RktSym">racket/gui</span></a></span> import earlier, we can supply <span class="RktWrap"><span class="RktVal">#f</span></span> here as an +argument to <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span>. Normally you supply the name of a shared +library file such as <span class="RktWrap"><span class="RktVal">"libcairo"</span></span>:</p> + +<div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"libcairo"</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"2"</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></p></div> + +<p>The last list argument specifies the accepted versions (<span class="RktWrap"><span class="RktVal">#f</span></span> allows +a version-less library). For this post, those details aren&rsquo;t important but +see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span> if you&rsquo;re curious.</p> + +<h1><a name="(part._.Extracting_functions)"></a>Extracting functions</h1> + +<p>Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> function. The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> +function takes three arguments:</p> + +<ul> + <li> + <p>The name of the value as a string (or symbol or bytestring)</p></li> + <li> + <p>a foreign library value, and</p></li> + <li> + <p>a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/types.html#%28tech._c._type%29"><span class="techinside">C type</span></a>, which is a type description that tells +the FFI how to marshall between Racket and C.</p></li></ul> + +<p>C types are a crucial concept for the FFI. They range from relatively +simple types like <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> to more complicated +type constructors such as <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span>. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span></span> with two functions that +handle marshalling between C and Racket code.</p> + +<p>To make progress with our Cairo code, we need to create a drawing context from +the surface object <span class="RktWrap"><span class="RktSym">bt-surface</span></span> that we defined a while ago. The +relevant function in the Cairo docs is +<a href="https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-create"><span class="stt">cairo_create</span></a>, +which has the following type signature:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">NB:</span><span class="hspace">&nbsp;</span><span class="RktCmt">this</span><span class="hspace">&nbsp;</span><span class="RktCmt">is</span><span class="hspace">&nbsp;</span><span class="RktCmt">C</span><span class="hspace">&nbsp;</span><span class="RktCmt">code</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_create</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_surface_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*target</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a <span class="stt">cairo_surface_t</span> and +returns a pointer to a <span class="stt">cairo_t</span>. Let&rsquo;s start with a very simple C type +that matches up with this behavior: </div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span></p></div></div> + +<div class="SIntrapara">This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type.</div> + +<p>The following definition shows how to use this type to obtain a foreign +function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Then we can use <span class="RktWrap"><span class="RktSym">cairo-create</span></span> as an ordinary racket function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">ctx</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Interlude__more_type_safety)"></a>Interlude: more type safety</h1> + +<p>Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> types, it is easy +to accidentally misuse the function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">You may not want to actually run this</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a cairo_t is not a cairo_surface_t</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>To prevent such bad uses, it is good practice to use <span class="emph">tagged</span> pointer types +using <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>. Here are two example definitions that +correspond to the <span class="stt">cairo_t</span> and <span class="stt">cairo_surface_t</span> types from earlier:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">The leading underscores are mandatory</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>We can then redefine <span class="RktWrap"><span class="RktSym">cairo-create</span></span> with a better type, which will +prevent ill-typed calls:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">cairo_surface_t-&gt;C: argument is not non-null</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">`cairo_surface_t' pointer</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktErr">argument: #&lt;cpointer:cairo_t&gt;</span></p></td></tr></tbody></table></div> + +<p>Unfortunately our old definition of <span class="RktWrap"><span class="RktSym">ctx</span></span> doesn&rsquo;t have this tag:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#f</span></p></td></tr></tbody></table></div> + +<p>Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it&rsquo;s also possible to update +existing pointers with a tag like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-push-tag%21%29%29">cpointer-push-tag!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#t</span></p></td></tr></tbody></table></div> + +<p>Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).</p> + +<h1><a name="(part._.Macros_for_reducing_boilerplate)"></a>Macros for reducing boilerplate</h1> + +<p>Now let&rsquo;s start building the FFI bindings for the functions in the Cairo sample. +First, let&rsquo;s go ahead and look at all of the types for the sample functions +from the C API docs:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_cap_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">line_cap</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Starting with <span class="stt">cairo_move_to</span>, we can set up a definition like we did +with <span class="stt">cairo_create</span> before:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">"cairo_move_to"</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span> library that help reduce the +verbosity. Here&rsquo;s an alternative definition of <span class="RktWrap"><span class="RktSym">cairo-move-to</span></span> +using the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form from +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>As you can see, the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +<span class="RktWrap"><span class="RktSym">cairo_move_to</span></span>) you also don&rsquo;t need to supply the C name either.</p> + +<p>The definitions for <span class="stt">cairo_line_to</span>, <span class="stt">cairo_set_line_width</span>, and +<span class="stt">cairo_stroke</span> aren&rsquo;t very interesting, so I&rsquo;ll just include them +below without comment:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="stt">cairo_set_line_cap</span> case is more interesting because the type +<span class="stt">cairo_line_cap_t</span> is a C enumeration type. Racket&rsquo;s FFI comes with +convenience forms for defining enumeration types&#8212; + <wbr />though it&rsquo;s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here&rsquo;s a quote from the Barzilay and Orlovsky +paper on that:</p> + +<blockquote class="SubFlow"> + <p>Our design follows a simple principle: keep C-level +functionality to a minimum.</p></blockquote> + +<p>and specifically about enumerations:</p> + +<blockquote class="SubFlow"> + <p>For example, the C level part of our interface does not commit to a +specific implementation for enumerations &#8212; it simply exposes C integers.</p></blockquote> + +<p>To define an enumeration, we can use the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the <span class="stt">cairo_line_cap_t</span> type, it suffices to +just supply the cases as a list of symbols:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It&rsquo;s also possible to specify how the symbols map to integers more precisely +(see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> for those details).</p> + +<p>Given this type, we can specify the type for the line cap function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<h1><a name="(part._.Putting_it_all_together)"></a>Putting it all together</h1> + +<p>Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">30.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-cap</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">round</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object <span class="RktWrap"><span class="RktSym">bt</span></span>. To see the +results of what we drew, we can just evaluate <span class="RktWrap"><span class="RktSym">bt</span></span> at the REPL. But it&rsquo;s +a little nicer if we use the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> library to draw a frame around +it to distinguish it from the background:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>And we&rsquo;re done! Of course, there is a lot more to the FFI. For example, I haven&rsquo;t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I&rsquo;m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!</p> \ No newline at end of file diff --git a/blog/feeds/tutorial.rss.xml b/blog/feeds/tutorial.rss.xml new file mode 100644 index 00000000..b47c8ef0 --- /dev/null +++ b/blog/feeds/tutorial.rss.xml @@ -0,0 +1,5283 @@ + + + + PRL Blog: Posts tagged 'tutorial' + PRL Blog: Posts tagged 'tutorial' + http://prl.ccs.neu.edu/blog/tags/tutorial.html + Sun, 17 Feb 2019 16:20:50 UT + Sun, 17 Feb 2019 16:20:50 UT + 1800 + + Writing a paper with Scribble + http://prl.ccs.neu.edu/blog/2019/02/17/writing-a-paper-with-scribble/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2019-02-17-writing-a-paper-with-scribble + Sun, 17 Feb 2019 16:20:50 UT + Ben Greenman + +<p>This post explains how to get started using Scribble to write a research paper.</p> +<!-- more--> + +<hr /> + +<blockquote> + <p>This post was written using <a href="http://download.racket-lang.org/all-versions.html">Racket 7.1</a> and <a href="https://github.com/racket/scribble/releases/tag/v7.1">Scribble 1.29</a></p></blockquote> + +<p>Writing about research is always difficult, but a compile-to-LaTeX tool can make the task easier. If your research code is written in the same language as the paper, then:</p> + +<ul> + <li>the paper can import definitions from the research, keeping a single point of control;</li> + <li>the language&rsquo;s functional abstractions can help manage the writing;</li> + <li>the language&rsquo;s drawing and/or plotting libraries can replace <a href="https://ctan.org/pkg/pgf?lang=en">TikZ</a>;</li> + <li>and you can write unit tests to validate the claims made in the paper.</li></ul> + +<p>Scribble, <a href="http://docs.racket-lang.org/scribble/index.html">the Racket documentation tool</a>, comes with a to-LaTeX compiler and a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> library tailored to the new <a href="https://ctan.org/pkg/acmart?lang=en">ACM paper format</a>. I have been a pretty happy user of these tools. In the interest of attracting more happy users, this post presents a short &ldquo;getting started&rdquo; guide and links to some larger examples.</p> + +<blockquote> + <p>For a Scribble tutorial, see the links in: <a href="/blog/2017/05/23/building-a-website-with-scribble/index.html">Building a Website with Scribble</a></p></blockquote> + +<h2 id="getting-started-with-">Getting started with <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a></h2> + +<p>The first line of a <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> document sets the formatting options (similar to a LaTeX file using <code>acmart.cls</code>). For example, the <a href="https://conf.researchr.org/track/gpce-2018/gpce-2018#Call-for-Papers">GPCE 2018 call for papers</a> asks for anonymized <code>sigplan</code>-format submissions with line numbers and 10 point font. The proper Scribble incantation is:</p> + +<pre><code>#lang scribble/acmart @sigplan @anonymous @review @10pt</code></pre> + +<p>Next, you may want to import some definitions. If we have a file <code>references.rkt</code> (see below for a definition), we can import it as follows:</p> + +<pre><code>@require{references.rkt}</code></pre> + +<p>The third main ingredient is the title and author information:</p> + +<pre><code>@(define neu (affiliation #:institution "Northeastern University")) +@(define anon (email "anon@anon.net")) + +@title{Writing a paper with Scribble} +@author[#:affiliation neu #:email anon]{Ben Greenman} + +@; optional: set the author names in the page headers +@elem[#:style "Sshortauthors"]{B. Greenman}</code></pre> + +<p>The paper is now ready to be written. You can forge ahead with a new <a href="http://docs.racket-lang.org/scribble/base.html#%28def._%28%28lib._scribble%2Fbase..rkt%29._section%29%29">section</a> and start adding content to the same file; alternatively, you can organize the writing across different modules. In this post, we will use the main document as an outline and <a href="http://docs.racket-lang.org/scribble/base.html#%28form._%28%28lib._scribble%2Fbase..rkt%29._include-section%29%29">import</a> content from other modules:</p> + +<pre><code>@include-abstract{abstract.scrbl} +@include-section{introduction.scrbl}</code></pre> + +<p>Finally, the main page is a good place to <a href="https://docs.racket-lang.org/scriblib/autobib.html">generate the bibliography</a>. Assuming this document imports a file like the <code>references.rkt</code> below, this expression inserts a bibliography titled &ldquo;References&rdquo;:</p> + +<pre><code>@generate-bibliography[#:sec-title "References"]</code></pre> + +<p>To build the document, invoke <code>scribble</code> on the command-line with the <code>--pdf</code> or <code>--latex</code> options:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl</code></pre> + +<p>If all goes well, this command generates a <code>FILE.pdf</code> with properly-linked cross references.</p> + +<h3 id="auxiliary-files">Auxiliary Files</h3> + +<p>If you save the code above to a file <code>example.scrbl</code> and save the files below in the same directory, then you should be able to build an <code>example.pdf</code>.</p> + +<p>These files are available in a slightly different format at this link:</p> + +<ul> + <li><a href="https://gitlab.com/bengreenman/scribble-acmart-example">https://gitlab.com/bengreenman/scribble-acmart-example</a></li></ul> + +<h4 id="referencesrkt"><code>references.rkt</code></h4> + +<pre><code>#lang racket/base + +(provide + ~cite citet generate-bibliography + fbf-icfp-2009) + +(require + scriblib/autobib) + +(define-cite ~cite citet generate-bibliography + #:style author+date-square-bracket-style) + +(define icfp "ICFP") + +(define fbf-icfp-2009 + (make-bib + #:title "Scribble: Closing the Book on Ad Hoc Documentation Tools" + #:author (authors "Matthew Flatt" "Eli Barzilay" "Robert Bruce Findler") + #:location (proceedings-location icfp #:pages '(109 120)) + #:date 2017))</code></pre> + +<h4 id="abstractscrbl"><code>abstract.scrbl</code></h4> + +<pre><code>#lang scribble/acmart + +A simple Scribble document.</code></pre> + +<h4 id="introductionscrbl"><code>introduction.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require{references.rkt} + +@; start with `title` instead of `section`, because importing via +@; `include-section` shifts all title/section/subsections down one level +@title{Introduction} + +Scribble creates a connection between a stand-alone document and the artifact +it describes@~cite[fbf-icfp-2009].</code></pre> + +<h3 id="q-how-to-debug-scribble-error-messages">Q. How to debug Scribble error messages?</h3> + +<p>If something goes wrong building a Scribble document, Racket is usually able to give a helpful error message.</p> + +<p>As a compile-time example, adding <code>@ foo</code> to a document produces the message <code>unexpected whitespace after @</code> and you can either delete the whitespace or change the <code>@</code> to <code>@"@"</code> for a literal <code>@</code>-sign.</p> + +<p>As a run-time example, adding <code>@(+ 2 2)</code> produces this message:</p> + +<pre><code>not valid in document body (need a pre-part for decode) in: 4</code></pre> + +<p>One fix is to convert <code>4</code> to a string, as in <code>@~a[(+ 2 2)]</code>.</p> + +<p>But if something goes wrong when Scribble renders a generated document to PDF, the default error output is <strong>not</strong> likely to help. For example, adding <code>@elem[#:style "oops"]</code> to a document produces a giant message:</p> + +<pre><code>$ raco scribble --pdf FILE.scrbl +[[ ... 84K of output ... ]] +Output written on example.pdf (1 page, 277876 bytes). +PDF statistics: + 53 PDF objects out of 1000 (max. 8388607) + 37 compressed objects within 1 object stream + 7 named destinations out of 1000 (max. 500000) + 36877 words of extra memory for PDF output out of 42996 (max. 10000000) + +run-pdflatex: got error exit code + context...: + [[ ... 17 more lines ... ]]</code></pre> + +<p>The best way to debug these messages is to <strong>ignore them</strong> and use a LaTeX compiler directly. For the &ldquo;oops&rdquo; mistake, LaTeX stops at the undefined control sequence &mdash; giving a hint about how to find the problem:</p> + +<pre><code>$ raco scribble --latex FILE.scrbl +$ pdflatex FILE.tex +[[ ... 12KB of output ... ]] +! Undefined control sequence. +l.549 \oops + {} +? </code></pre> + +<h3 id="q-how-to-add-a-latex-style-file">Q. How to add a LaTeX style file?</h3> + +<p>To add extra LaTeX code to the final document, create a new file and include it with the <code>++style</code> command-line flag. This copies the contents of the style file into the generated document (the copy appears near the top of the generated code).</p> + +<pre><code>$ raco scribble ++style style.tex --pdf FILE.scrbl</code></pre> + +<p>Here is an example style file.</p> + +<h4 id="styletex"><code>style.tex</code></h4> + +<pre><code>\settopmatter{printfolios=true,printccs=true,printacmref=true} +% add page numbers etc. + +\overfullrule=1mm +% draw a black rectangle near lines that overflow the margin</code></pre> + +<p>Another way to add extra LaTeX code is to add a <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Flatex-properties..rkt%29._tex-addition%29%29"><code>tex-addition</code></a> style property to the main title. This second approach makes it easy to include more than one file:</p> + +<pre><code>#lang scribble/acmart + +@require[ + (only-in scribble/core make-style) + (only-in scribble/latex-properties make-tex-addition)] + +@(define extra-style-files + (list (make-tex-addition "style.tex"))) + +@title[#:style (make-style #f extra-style-files)]{Writing a paper with Scribble} + +@; ....</code></pre> + +<h3 id="q-how-to-make-a-figure">Q. How to make a figure?</h3> + +<p>Use the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library to add figures to a document.</p> + +<pre><code>@require[pict scriblib/figure] +@figure[ + "fig:fish" @; figure tag, see `figure-ref` + @elem{A Standard Fish} @; figure caption, appears below the content + @elem{fish = @(standard-fish 90 40)}] @; content</code></pre> + +<p>The content of a figure can be almost anything that would work in the toplevel of the document.</p> + +<h3 id="q-how-to-include-extra-files-pictures-latex">Q. How to include extra files (pictures, LaTeX)?</h3> + +<p>The <code>++extra</code> command-line flag names an auxilliary file that Scribble should include when rendering the document. This flag may be supplied more than once.</p> + +<p>For example, if a document includes the content of an external LaTeX file:</p> + +<pre><code>@elem[#:style "input"]{inline-this.tex}</code></pre> + +<p>then make sure to build the document with a command like this:</p> + +<pre><code>$ raco scribble ++style style.tex ++extra inline-this.tex FILE.scrbl</code></pre> + +<h4 id="inline-thistex"><code>inline-this.tex</code></h4> + +<pre><code>% Raw LaTeX allowed here +$\lambda x.\, x$</code></pre> + +<h3 id="q-what-about-in-line-latex">Q. What about in-line LaTeX?</h3> + +<p>An <a href="https://docs.racket-lang.org/scribble/core.html#%28def._%28%28lib._scribble%2Fcore..rkt%29._element%29%29">element</a> with the <a href="https://docs.racket-lang.org/scribble/core.html#%28idx._%28gentag._60._%28lib._scribblings%2Fscribble%2Fscribble..scrbl%29%29%29"><code>'exact-chars</code></a> <a href="https://docs.racket-lang.org/scribble/core.html#%28tech._style._property%29">style property</a> renders directly to LaTeX.</p> + +<pre><code>@(define (exact . stuff) + @; the style name "relax" puts a `\relax` no-op in front of the stuff + (make-element (make-style "relax" '(exact-chars)) stuff)) + +@exact|{$\lambda x.\, x$}| +@; ==&gt; \relax{$\lambda x.\, x$} + +@(define ($ . math-stuff) + (apply exact (list "$" math-stuff "$"))) + +@${\lambda x.\, x} +@; ==&gt; \relax{$\lambda x.\, x$}</code></pre> + +<h2 id="creating-a-httpdocsracket-langorgguidemodulesyntaxhtml28parthash-lang29lang-for-a-paper">Creating a <a href="http://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29">#lang</a> for a paper</h2> + +<p>For a Scribble document that is split across multiple files, it can be helpful to make a <code>#lang</code> that <a href="http://blog.racket-lang.org/2017/03/languages-as-dotfiles.html">provides a common environment</a>. Instead of starting each file with a <code>require</code>, e.g.:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang scribble/acmart +@require["references.rkt" "helper-functions.rkt" scriblib/figure] + +....</code></pre> + +<p>files can start with a name that describes their common purpose:</p> + +<h4 id="paperscrbl"><code>paper.scrbl</code></h4> + +<pre><code>#lang conference-2018-submission + +....</code></pre> + +<p>As a bonus, if the language is defined as a package then the Scribble document can use Racket&rsquo;s dependency management tools:</p> + +<pre><code># to install the paper and interactively install dependencies: +$ cd conference-2018-submission; +$ raco pkg install + +# To check that the paper builds with no dependency issues: +$ raco setup --check-pkg-deps conference-2018-submission + +# To run all unit tests +$ raco test -c conference-2018-submission</code></pre> + +<p>To create a package and language:</p> + +<ol> + <li>Move the Scribble document to a directory with the language name, i.e., <code>conference-2018-submission/</code></li> + <li>Write a simple <code>info.rkt</code> to configure the package</li> + <li>Create a normal Racket module that exports the common environment</li> + <li>Create a <code>conference-2018-submission/lang/reader.rkt</code> module</li></ol> + +<p>Details below. For a full example, visit:</p> + +<ul> + <li><a href="https://gitlab.com/bennn/scribble-acmart-example">https://gitlab.com/bennn/scribble-acmart-example</a></li></ul> + +<hr /> + +<h4 id="conference-2018-submissioninforkt"><code>conference-2018-submission/info.rkt</code></h4> + +<p>This file defines the basic metadata for a package. For more about <code>info.rkt</code>, see: <a href="http://blog.racket-lang.org/2017/10/tutorial-creating-a-package.html">Tutorial: Creating a Package</a>.</p> + +<pre><code>#lang info +(define collection "conference-2018-submission") +(define deps '("base" "scribble-lib" "at-exp-lib")) +(define build-deps '("racket-doc" "scribble-doc")) +(define pkg-desc "Paper for Conference 2018") +(define version "0.1")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionmainrkt"><code>conference-2018-submission/main.rkt</code></h4> + +<p>This file defines and exports the common environment for every file in our Scribble document. In this example, the common environment is: the <a href="http://docs.racket-lang.org/scribble/ACM_Paper_Format.html">scribble/acmart</a> language, the file &ldquo;references.rkt&rdquo;, and the <a href="http://docs.racket-lang.org/scriblib/figure.html#%28def._%28%28lib._scriblib%2Ffigure..rkt%29._figure%29%29">scriblib/figure</a> library.</p> + +<pre><code>#lang racket/base + +(provide + (all-from-out + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")) + +(require + scribble/acmart + scribble/acmart/lang + scriblib/figure + "references.rkt")</code></pre> + +<br /> + +<h4 id="conference-2018-submissionlangreaderrkt"><code>conference-2018-submission/lang/reader.rkt</code></h4> + +<p>This file: (1) tells Racket to use the Scribble reader on <code>#lang conference-2018-submission</code> modules, and (2) wraps the result of such modules in a shape that Scribble expects.</p> + +<pre><code>#lang s-exp scribble/base/reader +conference-2018-submission +#:wrapper1 (lambda (t) (cons 'doc (t)))</code></pre> + +<h2 id="links-to-example-documents">Links to Example Documents</h2> + +<p>These documents use the <code>#lang</code> approach to writing a paper with Scribble. Check their <code>main.rkt</code> for example formatting functions and unit tests, and check the <code>.scrbl</code> files to see how the ideas above look in a larger document.</p> + +<ul> + <li><a href="https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018">https://github.com/nuprl/retic_performance/tree/master/gm-pepm-2018</a></li> + <li><a href="https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018">https://github.com/nuprl/tag-sound/tree/master/gf-icfp-2018</a></li></ul> + +<p>Finally, this repository provides a tool to start a new Scribble document:</p> + +<ul> + <li><a href="https://pkgd.racket-lang.org/pkgn/package/gtp-paper">https://pkgd.racket-lang.org/pkgn/package/gtp-paper</a></li></ul> + +<h2 id="further-reading">Further Reading</h2> + +<ul> + <li><a href="https://project.inria.fr/coqexchange/checking-machine-checked-proofs/">Checking Machine-Checked Proofs</a></li></ul> + + Defining Local Bindings in Turnstile Languages + http://prl.ccs.neu.edu/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-10-22-defining-local-bindings-in-turnstile-languages + Mon, 22 Oct 2018 15:05:17 UT + Sam Caldwell + +<p>In <a href="http://racket-lang.org/">Racket</a>, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Guide.html">Turnstile</a> is a new Racket {library,language} for creating typed languages by integrating type checking with Racket&rsquo;s existing tools for describing languages. The technique is described by fellow PRL&rsquo;ers in the paper <a href="http://www.ccs.neu.edu/home/stchang/pubs/ckg-popl2017.pdf"><em>Type Systems as Macros</em></a>.</p> + +<p>Racket encourages language developers to take full advantage of <a href="https://scholarship.rice.edu/handle/1911/17993">linguistic reuse</a> by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.</p> + +<p>Unfortunately, Turnstile does not lend itself to expressing one of Racket&rsquo;s most ubiquitous idioms: naming local bindings with <code>define</code>. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include <code>define</code>-like binding forms in their languages.</p> + +<p>This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a <code>define</code> binding form.</p> +<!-- more--> + +<p>The code for this blog post can be found <a href="https://gist.github.com/howell/e2d4501e24db503e4cd9aa368172a502">in this gist</a>. To run it, you will need the Turnstile package, which can be installed with <code>raco pkg install +turnstile</code>.</p> + +<h2 id="turnstile-typechecking-intertwined-with-elaboration">Turnstile: Typechecking Intertwined with Elaboration</h2> + +<p>Turnstile provides a convenient way of defining syntax transformations that also perform typechecking. Since processing the syntax of a form typically involves some amount of analysis, such as for error checking, it is a natural place to put the logic for typechecking. With forms defined as such, macro expanding a program determines both a type and an elaborated term in the target language.</p> + +<p>While macro expansion proceeds outside-in, type information typically flows up from the leaves of the AST during checking. To reconcile the two directions, Turnstile language forms invoke the macro expander on subexpressions when their types are needed for the current rule. This expansion yields both the elaboration of the term and its type, or fails with an error. Turnstile abstracts over the process of invoking the expander on subterms, allowing implementors to describe the language in terms of high-level type checking and elaboration specifications.</p> + +<h2 id="type--elaboration-rules">Type &amp; Elaboration Rules</h2> + +<p>To get a feel for defining language forms in Turnstile, this section walks through the core of a simply-typed functional language.</p> + +<h3 id="functions">Functions</h3> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-type-constructor</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="kd">#:arity</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e~3d))" style="color: inherit">&gt;=</a></span> <span class="mi">1</span><span class="p">)</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Looking at this item by item, we see:</p> + +<ol> + <li><code>define-type-constructor</code> creates a new type. Here, we say the <code>→</code> requires at least one parameter.</li> + <li><code>define-typed-syntax</code> is the primary way to define a language form in terms of its syntactic shape, how it is type checked, the target language term it expands to, and its type.</li> + <li>The next part is a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html">syntax-pattern</a> describing the the shape of the syntax this rule applies to. In this case, we&rsquo;re defining <code>λ</code> as a macro that expects a parenthesized sequence of identifier-colon-type triples, describing the formal arguments to the procedure, followed by the body <code>e</code>. The <code>type</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html">syntax class</a> is provided by Turnstile, and describes the surface syntax of types (such as those created with <code>define-type-constructor</code>); internal operations over types use the expanded version of the type, which is accessed via the <code>norm</code> attribute.</li> + <li>The chevron <code>≫</code> on the first line signifies that there is only one case in this type rule. Some rules, which we will see later, use multiple cases to check different kinds of uses.</li> + <li>The body of the rule is a sequence of premises, that usually check and analyze the types of sub-expressions, followed by a dashed line, and then the conclusion, describing the output syntax and its type.</li> + <li>Here, the single premise describes how to check the body of the function. The context, which associates variables with types, goes to the left of the turnstile (<code>⊢</code>). For each formal <code>x</code>, this lets us know what type <code>x</code> has when we find a reference to it in <code>e</code>. In this rule, we are saying &ldquo;while checking the right-hand-side, assume <code>x</code>&mdash;which elaborates to <code>x-</code>&mdash;has type <code>τ_in</code>, for each triple in the input syntax (signified by the ellipses <code>...</code>)&rdquo;. More on the &ldquo;elaborates to <code>x-</code>&rdquo; below.</li> + <li>To the right of the turnstile, we write the expression we are checking, <code>e</code>, and patterns <code>e-</code> and <code>τ_out</code> matching the elaboration of <code>e</code> and its type, respectively.</li> + <li>After the dashes comes the conclusion, which begins with <code>⊢</code>. The next part specifies the elaboration of the term. Here, the meaning of the typed <code>λ</code> is given in terms of Racket&rsquo;s <a href="http://docs.racket-lang.org/reference/lambda.html?q=%23%25plain-lambda#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~23~25plain-lambda%29%29"><code>#%plain-lambda</code></a>. Turnstile uses the convention of a <code>-</code> suffix for forms in the untyped/target language to avoid conflicting names and confusion. Suffixed names are usually bound using <code>postfix-in</code>, such as in <code>(require (postfix-in - racket/base))</code> to bind <code>#%plain-lambda-</code>.</li> + <li>Finally, we give the type of the term to the right of the <code>⇒</code>, referring to pattern variables bound in the premises.</li></ol> + +<h4 id="renaming-typed-variables">Renaming Typed Variables</h4> + +<p>Turnstile lets the Racket expander take care of the details of variable scope, shadowing, etc. To associate identifier <code>x</code> with type <code>τ</code>, Turnstile binds <code>x</code> to a macro that knows <code>τ</code> when it expands. References to <code>x</code> now become references to that macro, and expanding them provides access to <code>τ</code>. Concretely, the underlying Racket code implementing this behavior looks roughly like this:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let-syntax))" style="color: inherit">let-syntax</a></span> <span class="p">([</span><span class="n">x</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._make-rename-transformer))" style="color: inherit">make-rename-transformer</a></span> <span class="n">x-</span><span class="p">)])</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="nb"><a href="http://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#(def._((quote._~23~25kernel)._expand))" style="color: inherit">expand</a></span> <span class="k"><a href="http://docs.racket-lang.org/reference/if.html#(form._((lib._racket/private/letstx-scheme..rkt)._and))" style="color: inherit">and</a></span> <span class="n">check</span> <span class="n">forms</span> <span class="n">that</span> <span class="n">may</span> <span class="n">reference</span> <span class="n">x</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The type <code>τ</code> is attached as <a href="http://docs.racket-lang.org/reference/stxprops.html">metadata</a> for a new identifier <code>x-</code>, which is what <code>x</code> will transform to at any reference site. In order for this to work, <code>x-</code> must be distinct from <code>x</code>&mdash;hence the <code>generate-temporary</code>&mdash;to avoid an infinite expansion loop.</p> + +<h3 id="application">Application</h3> + +<p>We can define a version of <code>#%app</code> that type checks function applications to accompany our typed <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/application.html#(form._((lib._racket/private/base..rkt)._~23~25app))" style="color: inherit">#%app</a></span> <span class="n">e_fn</span> <span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_fn</span> <span class="n">≫</span> <span class="n">e_fn-</span> <span class="n">⇒</span> <span class="p">(</span><span class="n">~→</span> <span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)]</span> + <span class="kd">#:fail-unless</span> <span class="p">(</span><span class="n">stx-length=?</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">(</span><span class="n">num-args-fail-msg</span> <span class="o">#'</span><span class="n">e_fn</span> <span class="o">#'</span><span class="p">[</span><span class="n">τ_in</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">]</span> <span class="o">#'</span><span class="p">[</span><span class="n">e_arg</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">])</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e_arg</span> <span class="n">≫</span> <span class="n">e_arg-</span> <span class="n">⇐</span> <span class="n">τ_in</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">--------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-app-</span> <span class="n">e_fn-</span> <span class="n">e_arg-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<ol> + <li>The syntax pattern on the first line describes the shape of applications.</li> + <li>On the second line, we pattern match the result of expanding and checking <code>e_fn</code>, checking that it produces an arrow type. More specifically, when we defined the arrow type <code>→</code> above, <code>define-type-constructor</code> also implicitly defined a <a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html?q=pattern%20expander#%28part._.Pattern_.Expanders%29">pattern expander</a> <code>~→</code> (which uses the Racket <code>~</code> prefix convention for syntax patterns) that matches instances of the type.</li> + <li>The next clause checks that the number of provided arguments matches the arity of the function as specified by its type.</li> + <li>Line 5 checks that each argument expression has the required type. Turnstile uses <a href="http://davidchristiansen.dk/tutorials/bidirectional.pdf">bidirectional typechecking rules</a>, which either infer the type of a term or checks that a term satisfies a given type. We write <code>⇐ τ_in</code> in the premise to switch to checking mode.</li> + <li>Finally, typed function application elaborates to Racket&rsquo;s function application, <code>#%plain-app</code>, with the usual suffix, and produces type <code>τ_out</code> for the application</li></ol> + +<p>We can try out these new typed forms on a few examples:</p> + +<ul> + <li><code>((λ ([x : Int]) (+ x 1)) 2)</code> successfully typechecks and yields <code>3</code>.</li> + <li><code>((λ ([x : Int]) (+ x 1)))</code> raises an error based on the check on lines 3 and 4 in the rule: "#%app: (λ ((x : Int)) (+ x 1)): wrong number of arguments: expected 1, given 0."</li> + <li><code>((λ ([x : (→ Int Int)]) (x 1)) 2)</code> raises an error: "#%app: type mismatch: expected (→ Int Int), given Int" as a consequence of using checking mode on line 5 of the rule.</li></ul> + +<h2 id="extending-our-language-with-local-bindings">Extending Our Language with Local Bindings</h2> + +<p>When writing functional programs, we often want to name various sub-computations. One way to do that is with a <code>let</code> construct, which Turnstile allows us to easily create:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="n">e-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e-x</span> <span class="n">≫</span> <span class="n">e-x-</span> <span class="n">⇒</span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ-x</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="n">e-body</span> <span class="n">≫</span> <span class="n">e-body-</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">]</span> + <span class="n">-------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">let-</span> <span class="p">([</span><span class="n">x-</span> <span class="n">e-x-</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-body-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-body</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unsurprisingly, this looks very similar to the definition of <code>λ</code> above. Now we can write functions with named intermediate results:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/let.html#(form._((lib._racket/private/letstx-scheme..rkt)._let))" style="color: inherit">let</a></span> <span class="p">([</span><span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)])</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>However, in Racket it&rsquo;s common to name such intermediate results using <code>define</code> rather than <code>let</code>. In fact, it&rsquo;s <a href="https://docs.racket-lang.org/style/Choosing_the_Right_Construct.html#%28part._.Definitions%29">prescribed by the style guide</a>. Naturally, we would like to do so in our Racket language extension as well, which would allow us to write the above function as:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost-there</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost-there</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, this is not nearly as easy to do in Turnstile as <code>let</code>.</p> + +<h2 id="sequences">Sequences</h2> + +<p>At first glance, the issue seems to be that the definition of <code>λ</code> above limits the body to be a single expression when what we want to put there is a sequence of definitions and expressions. To reach our goal, we need to change the definition of <code>λ</code> to allow its body to be a sequence.</p> + +<p>The first step is to create a typed form for sequences of definitions and expressions, which can then be used by rules like <code>λ</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="n">stx-last</span> <span class="o">#'</span><span class="p">(</span><span class="n">τ</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))</span> + <span class="n">-----------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="n">e-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This directs type checking to:</p> + +<ol> + <li>Check each <code>e</code> in the sequence individually, obtaining an expanded <code>e-</code> and inferred type <code>τ</code> for each.</li> + <li>Take the last type in the sequence and call it <code>τ-final</code>; Turnstile allows using <code>syntax-parse</code> <a href="http://docs.racket-lang.org/syntax/stxparse-specifying.html?#%28tech._pattern._directive%29">directives</a> such as <code>#:with</code> as premises.</li> + <li>Expand to Racket&rsquo;s <code>begin</code> (with the usual <code>-</code> suffix) and give the whole expression the type of the last term in the body.</li></ol> + +<p>Now, we can use <code>begin</code> in a revised definition of <code>λ</code>. The new rule takes a non-empty sequence of forms in the body and wraps them in our new <code>begin</code> form for typechecking.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x:id</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ_in:type</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[[</span><span class="n">x</span> <span class="n">≫</span> <span class="n">x-</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ_in.norm</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">⊢</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ_out</span><span class="p">]</span> + <span class="n">-------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">#%plain-lambda-</span> <span class="p">(</span><span class="n">x-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types-extra..rkt)._~e2~86~92))" style="color: inherit">→</a></span> <span class="n">τ_in.norm</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">τ_out</span><span class="p">)])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Now we need a way to include definitions in these sequences and we&rsquo;re set!</p> + +<h2 id="the-difficulty-with-define">The Difficulty With Define</h2> + +<p>If we think about how type information is communicated between a binder and its reference we can see why <code>define</code> is a different beast than <code>let</code></p> + +<pre><code>(let ([x 5]) (+ x 1)) + ^ ^ + | |- TO HERE +FROM HERE</code></pre> + +<p>When the rule for our <code>let</code> is invoked, it has access to both the binding sites and the place where references may occur. The situation lends itself to a straightforward implementation strategy: create an environment of identifier/type associations to use when analyzing the body. Turnstile directly accommodates this scenario in its language for creating type rules with the optional context appearing on the left of the <code>⊢</code>, as in our rules for <code>λ</code> and <code>let</code> above.</p> + +<p>Define is different.</p> + +<pre><code>(define x 5) + ^ + |------ TO WHERE? +FROM HERE</code></pre> + +<p>The problem is apparent: we can&rsquo;t see where the reference to <code>x</code> occurs! The information about the binding needs to escape from the <code>define</code> to the surrounding context. In other words, when we implement <code>define</code>, we don&rsquo;t have a body term available that contains all the possible references. Instead, we will have to find a way of communicating the existence of the <code>x</code> binding and its type to the surrounding context.</p> + +<p>Above, in the subsection on &ldquo;Renaming Typed Variables&rdquo;, we saw that the context in Turnstile type rules is implemented as syntax transformers with <code>let</code>-like scope (created with <code>let-syntax</code>). One idea would be to mimic this approach, but instead of using <code>let-syntax</code> to achieve <code>let</code>-like scope, use <code>define-syntax</code> to achieve <code>define</code>-like scope.</p> + +<p>Fortunately for us, someone has already tried their hand at writing a <code>define</code> form for Turnstile languages using a <code>define-syntax</code> rename, found in the <a href="https://github.com/stchang/macrotypes/blob/c5b663f7e663c564cb2baf0e0a352d5fde4d2bd7/turnstile/examples/ext-stlc.rkt#L55">Turnstile examples</a>. We can take that as our starting point:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-base-type</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">)</span> +<span class="p">(</span><span class="n">define-</span> <span class="n">a-deep-dark-void</span> <span class="p">(</span><span class="n">#%app-</span> <span class="n">void-</span><span class="p">))</span> +<span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n">assign-type</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="n">-----------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e-</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> + <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Let&rsquo;s break it down.</p> + +<ol> + <li>Create a new type, <code>Void</code>, to assign definitions.</li> + <li>Create a constant to serve as the canonical value of type <code>Void</code>.</li> + <li>Define a new typed form, <code>define</code>, used as in <code>(define x e)</code>.</li> + <li>Check the type of the expression <code>e</code>, getting its expansion <code>e-</code> and type <code>τ</code>.</li> + <li>Create a new name, <code>x-</code>, and attach the type <code>τ</code> as metadata.</li> + <li>Expand to Racket&rsquo;s <code>begin</code>. Unlike <code>let</code>, <code>begin</code> does not create a new scope; definitions inside a <code>begin</code> are also visible in the surrounding context. That behavior is needed for scenarios like this one that expand to multiple definitions.</li> + <li>Create a macro binding for <code>x</code> that rewrites to <code>x-</code>. By using a define-like form, the macro has the same scoping rules as <code>define</code>, so it will apply to references to <code>x</code> in the surrounding context&mdash;exactly what we want. (We are using <code>make-variable-like-transformer</code> to avoid the special treatment the expander gives to <code>rename-transformer</code>s. The specifics are beyond the scope of this post.)</li> + <li>Define <code>x-</code> to refer to the supplied expression. Note that here <code>define-</code> is Racket&rsquo;s <code>define</code>.</li> + <li>Keep the result of evaluating this form in line with the type by yielding a value of type <code>Void</code>.</li></ol> + +<p>This implementation of <code>define</code> gets us pretty far. If we put definitions at the top-level of a module in our language, we can reference them within other terms in the module:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="c1">;; module top level</span> +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x</span> <span class="mi">5</span><span class="p">)</span> +<span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">)</span> +<span class="c1">;;=&gt; 6</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Unfortunately, we encounter a problem if we try to create <em>local</em> definitions:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">add2</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/lambda.html#(form._((lib._racket/private/base..rkt)._~ce~bb))" style="color: inherit">λ</a></span> <span class="p">([</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">)))</span> +<span class="c1">;;==&gt; almost: unbound identifier...</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Pointing to the reference on the final line. The problem is that our <code>define</code> and <code>begin</code> forms are not interacting in the way we might have hoped.</p> + +<p>When we expand the body of the function above, we associate <code>x</code> with type <code>Int</code> then start checking the body, wrapped in a <code>begin</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Consulting the definition of <code>begin</code>, we see that it checks/expands each sub-expression in seqence. First in the sequence is a use of <code>define</code>, yielding</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Crucially, the expansion of our <code>define</code> form <strong>stops</strong> at this point, without examining the <code>begin-</code> form and its contained definitions. The interface through which Turnstile invokes the macro expander, <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>, takes a parameter referred to as the <em>stop list</em> for stopping expansion at certain points. The stop list contains identifiers which, when encountered by the expander, halt expansion.</p> + +<p>The syntax output from typed forms created using Turnstile are wrapped with a particular macro, named <code>erased</code>, that serves (only) to orchestrate stopping expansion. So, the output of our <code>define</code> form actually looks like</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">erased</span> + <span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">almost</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">almost-</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And since Turnstile includes <code>erased</code> in the stop list for <code>local-expand</code>, expansion stops before analyzing the rest of the output. The point of all this <code>erased</code> business, if you are wondering, is to improve the performance of Turnstile languages by avoiding unnecessary re-expansions.</p> + +<p>Control returns to the <code>begin</code> transformer, which turns to checking/expanding the subsequent <code>(+ almost 1)</code>, where it will encounter the identifier <code>almost</code> without a corresponding binding. Even though our <code>define</code> form produced a binding as part of its output, the expander hasn&rsquo;t actually analyzed it before reaching the reference in the next expression.</p> + +<p>The problem is a symptom of analyzing the sequence of forms using an ellipses, which corresponds to mapping the typechecking/expanding process over each individually. The mapping operation stipulates that checking each item is independent of checking the others. But when we add <code>define</code> to the language that is no longer the case. A definition form influences how we typecheck its neighbors by introducing a new name and its type. This information must be communicated to the following forms in order to properly check references. That is, instead of setting up binding information and then checking, analyzing bindings must be interleaved with type checking. Unfortunately, Turnstile doesn&rsquo;t provide a fold-like mechanism for threading binding information through the checking of a sequence of typed forms. We&rsquo;re going to need to implement our own solution, requiring us to dive underneath the abstractions provided by Turnstile and get intimate with Racket&rsquo;s syntax model.</p> + +<h2 id="internal-definition-contexts">Internal Definition Contexts</h2> + +<p>In order for the <code>(+ almost 1)</code> expression from above to successfully typecheck/expand, we need to be able to associate <code>almost</code> with a suitable type. Turnstile provides a way to set up such an association, but as we saw before, Turnstile&rsquo;s interface doesn&rsquo;t suit this scenario.</p> + +<p>Racket has the notion of an <a href="http://docs.racket-lang.org/reference/syntax-model.html?#%28tech._internal._definition._context%29">internal definition context</a> that allows definitions to be mixed with expressions. The syntax system exposes tools for creating and manipulating such contexts programmatically, allowing macro writers a great deal of power for manipulating the bindings in a program.</p> + +<p>When using <code>local-expand</code>, we can optionally pass in a definition context containing binding information. If we create a definition context for the body of the function and extend it with each definition, then <code>local-expand</code>-ing references such as the above one should work out. Normally, Turnstile calls <code>local-expand</code> internally in accordance with the type rules we write down, but in order to use our own definition context we&rsquo;re going to have to call it ourselves.</p> + +<p>We can create a definition context with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-make-definition-context%29%29"><code>syntax-local-make-definition-context</code></a>, as in</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And then (imperatively) add bindings to <code>def-ctx</code> with <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-bind-syntaxes%29%29"><code>syntax-local-bind-syntaxes</code></a>. The first argument is a list of identifiers to bind; we will only be binding one identifier at a time, consequently only passing singleton lists. The second argument dictates what the given identifier <em>means</em>. Passing <code>#f</code> corresponds to a run-time/phase 0 binding, such as that of a procedure argument, <code>let</code>, or <code>define</code>; alternatively, we can provide syntax that evaluates to a function, establishing a transformer binding invoked on references to the identifier. Using both alternatives, we can define a renaming macro and give a meaning to the new name:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="n">x</span> <span class="n">x-</span> <span class="n">t</span> <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x</span><span class="p">)</span> + <span class="o">#`</span><span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> + <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'#,</span><span class="n">x-</span> <span class="o">#'#,</span><span class="n">t</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">))</span> + <span class="n">ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-bind-syntaxes))" style="color: inherit">syntax-local-bind-syntaxes</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">x-</span><span class="p">)</span> <span class="no">#f</span> <span class="n">ctx</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The first call binds <code>x</code> to a transformer that renames to <code>x-</code>; the second lets the expander know that we are taking care of making sure that <code>x-</code> will actually be bound to something.</p> + +<p>Our <code>define</code> form must communicate the information needed to call <code>int-def-ctx-bind-type-rename!</code> back out to the surrounding context. One way to do this is to add an intermediate step to the expansion of <code>define</code> that includes the necessary information as part of its syntax. Then, the surrounding context can analyze the expansion of each term, looking for that form.</p> + +<p>Concretely, <code>define</code> will expand to <code>define/intermediate</code>, which will in turn expand to what <code>define</code> originally expanded to:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">x:id</span> <span class="n">e</span><span class="p">)</span> <span class="n">≫</span> + <span class="p">[</span><span class="n">⊢</span> <span class="n">e</span> <span class="n">≫</span> <span class="n">e-</span> <span class="n">⇒</span> <span class="n">τ</span><span class="p">]</span> + <span class="kd">#:with</span> <span class="n">x-</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/reference/syntax-util.html#(def._((lib._racket/syntax..rkt)._generate-temporary))" style="color: inherit">generate-temporary</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x+</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-identifier-as-binding))" style="color: inherit">syntax-local-identifier-as-binding</a></span> <span class="o">#'</span><span class="n">x</span><span class="p">)</span> + <span class="n">--------------------------------------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x+</span> <span class="n">x-</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">)</span> <span class="n">⇒</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/type-ref.html#(form._((lib._typed-racket/base-env/base-types..rkt)._.Void))" style="color: inherit">Void</a></span><span class="p">])</span> + +<span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e</span><span class="p">)</span> + <span class="kd">#:with</span> <span class="n">x-/τ</span> <span class="p">(</span><span class="n">assign-type</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="kd">#:wrap?</span> <span class="no">#f</span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="n">begin-</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/transformer-helpers.html#(def._((lib._syntax/transformer..rkt)._make-variable-like-transformer))" style="color: inherit">make-variable-like-transformer</a></span> <span class="o">#'</span><span class="n">x-/τ</span><span class="p">))</span> + <span class="p">(</span><span class="n">define-</span> <span class="n">x-</span> <span class="n">e</span><span class="p">)</span> + <span class="n">a-deep-dark-void</span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>(The reason we create an <code>x+</code> using <a href="http://docs.racket-lang.org/reference/stxtrans.html#%28def._%28%28quote._~23~25kernel%29._syntax-local-identifier-as-binding%29%29"><code>syntax-local-identifier-as-binding</code></a> is <a href="https://github.com/racket/racket/pull/2237">due to a bug in the expander</a>. The explanation is rather involved and frankly I only barely understand what&rsquo;s going on myself (if at all), so let&rsquo;s just leave it at that and move on.)</p> + +<p>Then, for each form <code>e</code> in a sequence, we can call <code>local-expand</code> with <code>def-ctx</code> and then check the expansion, <code>e-</code>, for <code>define/intermediate</code>. In those cases, we can use <code>int-def-ctx-bind-type-rename!</code> to add it to the context. The procedure <code>add-bindings-to-ctx!</code> performs this check on an expanded form <code>e-</code> (remembering that Turnstile will wrap the output of <code>define</code> in an <code>erased</code> macro):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">e-</span> + <span class="kd">#:literals</span> <span class="p">(</span><span class="n">erased</span><span class="p">)</span> + <span class="p">[(</span><span class="n">erased</span> <span class="p">(</span><span class="n">define/intermediate</span> <span class="n">x:id</span> <span class="n">x-:id</span> <span class="n">τ</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">int-def-ctx-bind-type-rename!</span> <span class="o">#'</span><span class="n">x</span> <span class="o">#'</span><span class="n">x-</span> <span class="o">#'</span><span class="n">τ</span> <span class="n">def-ctx</span><span class="p">)]</span> + <span class="p">[</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> + <span class="c1">;; when e expands to something other than a definition there&#39;s nothing to bind</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/void.html#(def._((quote._~23~25kernel)._void))" style="color: inherit">void</a></span><span class="p">)]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>We now have the key ingredients to define a procedure, <code>walk/bind</code>, that will serve as the primary vehicle to type check a sequence of forms, threading binding information through using a definition context. Processing sequences of defintions and expressions will iterate through them one at a time, and for each form <code>e</code>:</p> + +<ol> + <li><code>local-expand</code> using our internal definition context, resulting in an <code>e-</code>.</li> + <li>Retrieve the type of <code>e</code> from the metadata of <code>e-</code> using Turnstile&rsquo;s <a href="http://docs.racket-lang.org/turnstile/The_Turnstile_Reference.html?q=typeof#%28def._%28%28lib._turnstile%2Fmain..rkt%29._typeof%29%29"><code>typeof</code></a> helper.</li> + <li>Check if <code>e</code> defined a binding, in which case add it to the context.</li></ol> + +<p>Aggregating the expanded syntax and type of each form as we go along, we get</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-for-syntax))" style="color: inherit">define-for-syntax</a></span> <span class="p">(</span><span class="n">walk/bind</span> <span class="n">e...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">def-ctx</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._syntax-local-make-definition-context))" style="color: inherit">syntax-local-make-definition-context</a></span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">unique</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/symbols.html#(def._((quote._~23~25kernel)._gensym))" style="color: inherit">gensym</a></span> <span class="o">'</span><span class="ss">walk/bind</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">rev-e-...</span> <span class="n">rev-τ...</span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/for.html#(form._((lib._racket/private/base..rkt)._for/fold))" style="color: inherit">for/fold</a></span> <span class="p">([</span><span class="n">rev-e-...</span> <span class="o">'</span><span class="p">()]</span> + <span class="p">[</span><span class="n">rev-τ...</span> <span class="o">'</span><span class="p">()])</span> + <span class="p">([</span><span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/sequences.html#(def._((lib._racket/sequence..rkt)._in-syntax))" style="color: inherit">in-syntax</a></span> <span class="n">e...</span><span class="p">)])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">e-</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/stxtrans.html#(def._((quote._~23~25kernel)._local-expand))" style="color: inherit">local-expand</a></span> <span class="n">e</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="n">unique</span><span class="p">)</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._list))" style="color: inherit">list</a></span> <span class="o">#'</span><span class="n">erased</span><span class="p">)</span> <span class="n">def-ctx</span><span class="p">))</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">τ</span> <span class="p">(</span><span class="n">typeof</span> <span class="n">e-</span><span class="p">))</span> + <span class="p">(</span><span class="n">add-bindings-to-ctx!</span> <span class="n">e-</span> <span class="n">def-ctx</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">e-</span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((quote._~23~25kernel)._cons))" style="color: inherit">cons</a></span> <span class="n">τ</span> <span class="n">rev-τ...</span><span class="p">))))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/values.html#(def._((quote._~23~25kernel)._values))" style="color: inherit">values</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-e-...</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/private/list..rkt)._reverse))" style="color: inherit">reverse</a></span> <span class="n">rev-τ...</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The value <code>unique</code> and its use as an argument is dictated by the documentation of <a href="http://docs.racket-lang.org/reference/stxtrans.html?q=local-expand#%28def._%28%28quote._~23~25kernel%29._local-expand%29%29"><code>local-expand</code></a>: &ldquo;For a particular internal-definition context, generate a unique value and put it into a list for context-v.&rdquo; By using <code>#'erased</code> in the stop list for <code>local-expand</code>, we stop expansion at the same points that Turnstile does.</p> + +<p>Now we can implement <code>begin</code> in terms of <code>walk/bind</code>:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="n">define-typed-syntax</span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> <span class="n">≫</span> + <span class="kd">#:do</span> <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((quote._~23~25kernel)._define-values))" style="color: inherit">define-values</a></span> <span class="p">(</span><span class="n">e-...</span> <span class="n">τ...</span><span class="p">)</span> <span class="p">(</span><span class="n">walk/bind</span> <span class="o">#'</span><span class="p">(</span><span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)))]</span> + <span class="kd">#:with</span> <span class="n">τ-final</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._last))" style="color: inherit">last</a></span> <span class="n">τ...</span><span class="p">)</span> + <span class="n">--------------------</span> + <span class="p">[</span><span class="n">⊢</span> <span class="p">(</span><span class="n">begin-</span> <span class="o">#,@</span><span class="n">e-...</span><span class="p">)</span> <span class="n">⇒</span> <span class="n">τ-final</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>and voilà!</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">add2</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">Int</span><span class="p">])</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">almost</span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">x</span> <span class="mi">1</span><span class="p">))</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._+))" style="color: inherit">+</a></span> <span class="n">almost</span> <span class="mi">1</span><span class="p">))</span> + +<span class="p">(</span><span class="n">add2</span> <span class="mi">3</span><span class="p">)</span> +<span class="c1">;;=&gt; 5</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="but-wait-theres-more">But Wait, There&rsquo;s More</h1> + +<p>I believe this design is can be dropped in &lsquo;as-is&rsquo; and with a few extensions be useful for a wide variety of Turnstile languages. However, there are a few shortcomings (that I am aware of) that I will leave as exercises for the interested reader:</p> + +<ul> + <li>The <code>define</code> form here doesn&rsquo;t provide the useful shorthand for creating functions, <code>(define (f x) e ...)</code>. Extending it to do so is relatively straightforward.</li> + <li>Supporting <em>recursive</em> (and mutually recursive) function definitions is a bit more complicated, but shouldn&rsquo;t require many changes to the above code.</li> + <li>There&rsquo;s an extensibility issue&mdash;macros that expand to multiple uses of <code>define</code> inside a <code>begin</code> won&rsquo;t work (why not?), such as</li></ul> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define-syntax))" style="color: inherit">define-syntax</a></span> <span class="p">(</span><span class="n">define/memo</span> <span class="n">stx</span><span class="p">)</span> + <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/Parsing_Syntax.html#(form._((lib._syntax/parse..rkt)._syntax-parse))" style="color: inherit">syntax-parse</a></span> <span class="n">stx</span> + <span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt).__))" style="color: inherit">_</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="p">(</span><span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._~7edatum))" style="color: inherit">~datum</a></span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span><span class="p">)</span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> <span class="n">e</span> <span class="n"><a href="http://docs.racket-lang.org/syntax/stxparse-patterns.html#(form._((lib._syntax/parse..rkt)._......+))" style="color: inherit">...+</a></span><span class="p">)</span> + <span class="o">#'</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/begin.html#(form._((quote._~23~25kernel)._begin))" style="color: inherit">begin</a></span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="n">memo</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">f</span> <span class="p">[</span><span class="n">x</span> <span class="n"><a href="http://docs.racket-lang.org/ts-reference/special-forms.html#(form._((lib._typed-racket/base-env/prims..rkt)._~3a))" style="color: inherit">:</a></span> <span class="n">τ</span><span class="p">]</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">)</span> + <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> <span class="n">check</span> <span class="n">memo</span> <span class="n">table</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span> + <span class="n">e</span> <span class="k"><a href="http://docs.racket-lang.org/reference/stx-patterns.html#(form._((lib._racket/private/stxcase-scheme..rkt)._......))" style="color: inherit">...</a></span><span class="p">))]))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Finally, there&rsquo;s some question as to how to lift these ideas to an abstraction at the Turnstile level, so that future language authors don&rsquo;t have to muck around with <code>syntax-local-bind-syntaxes</code> and friends. If you have any ideas on this front, feel free to reach out.</p> +<!-- References--> + + PLT Redex FAQ + http://prl.ccs.neu.edu/blog/2017/09/25/plt-redex-faq/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-09-25-plt-redex-faq + Mon, 25 Sep 2017 23:39:16 UT + Ben Greenman, Sam Caldwell + +<p>A short guide to Redex concepts, conventions, and common mistakes.</p> +<!-- more--> + +<hr /> + +<p><em>To contribute to this FAQ, submit issues and pull requests to: <a href="https://github.com/nuprl/website/">https://github.com/nuprl/website/</a></em></p> + +<h3 id="q-what-is-redex-useful-for">Q. What is Redex useful for?</h3> + +<ol> + <li>declaring <a href="https://en.wikipedia.org/wiki/Regular_tree_grammar">regular tree grammars</a></li> + <li>defining <em>pattern</em>-based judgments and relations on <em>terms</em></li> + <li>testing properties of the above</li></ol> + +<p>More generally, Redex is helpful for experimenting with a programming language design, and helping you decide what you might want to prove about a language.</p> + +<h3 id="q-what-is-redex-not-useful-for">Q. What is Redex <strong>not</strong> useful for?</h3> + +<p>Proving theorems about a grammar, judgment, or relation.</p> + +<h3 id="q-what-is-a-term">Q. What is a <em>term</em>?</h3> + +<p>Informally, a term is:</p> + +<ul> + <li>a Redex &ldquo;atom&rdquo;, or</li> + <li>an object that represents a sequence of characters.</li></ul> + +<p>More formally, a term is the result of evaluating <strong>(term X)</strong>, where <strong>X</strong> is any syntactically-correct Racket expression.</p> + +<p>Examples:</p> + +<pre><code>$ racket +Welcome to Racket v6.10.0.3. +&gt; (require redex/reduction-semantics) +&gt; (term 42) +42 +&gt; (term (+ 2 2)) +'(+ 2 2) +&gt; (term ("hello" world (#false))) +'("hello" world (#f))</code></pre> + +<p>Some terms may look strange. That&rsquo;s OK, because a term by itself has no meaning.</p> + +<p>Terms can refer to previously-defined values using the <strong>unquote</strong> escape.</p> + +<pre><code>&gt; (define x (term 42)) +&gt; (term (+ 2 x)) +'(+ 2 x) +&gt; (term (+ ,x (unquote x))) +'(+ 42 42)</code></pre> + +<h3 id="q-what-is-a-redex-model">Q. What is a <em>Redex model</em>?</h3> + +<p>A Redex model is collection of tools for working with terms. The tools may include:</p> + +<ul> + <li><em>languages</em>, to define a grammar for terms</li> + <li><em>judgments</em>, to describe properties of terms or relations between terms</li> + <li><em>metafunctions</em>, to transform terms</li> + <li><em>reduction relations</em>, to define a term rewriting system</li></ul> + +<p>The goal of these tools is to encode a &ldquo;real thing&rdquo; (maybe, a programming language) using Redex terms.</p> + +<h3 id="q-what-is-a-language">Q. What is a language?</h3> + +<p>A Redex <em>language</em> is a named set of non-terminals, <em>patterns</em>, and <em>binding forms</em>. For example, a Redex model of the natural numbers might start with this language:</p> + +<pre><code>(define-language nat + [N ::= Zero + (Plus1 N)])</code></pre> + +<ul> + <li>the name of the language is <strong>nat</strong></li> + <li>the non-terminal <strong>N</strong> is associated with two patterns: <strong>Zero</strong> and <strong>(Plus1 N)</strong></li> + <li>there are no <em>binding forms</em></li></ul> + +<p>Each pattern describes a syntactic category of terms. Each non-terminal gives a name to the union of the patterns that follow it.</p> + +<p>The non-terminal <strong>N</strong> describes all terms that are either:</p> + +<ol> + <li>the symbol <strong>Zero</strong></li> + <li>lists of the form <strong>(Plus1 N)</strong>, where <strong>N</strong> is either <strong>Zero</strong> or another &ldquo;Plus1&rdquo;</li></ol> + +<p>For example,</p> + +<pre><code>(term Zero) +(term (Plus1 Zero)) +(term (Plus1 (Plus1 Zero))) +(term (Plus1 (Plus1 (Plus1 Zero)))) +;; .... and so on</code></pre> + +<p>If a language has binding forms, then some terms can introduce names. See the question on <em>binding forms</em> (below) for an example.</p> + +<h3 id="q-what-is-a-pattern">Q. What is a pattern?</h3> + +<p>A pattern is a sequence of characters and variables. If you have: (1) a language, and (2) a pattern that contains <em>named non-terminals</em> from the language, then you can ask whether a Redex term matches the pattern.</p> + +<p>A <em>named non-terminal</em> for a language <strong>L</strong> is an identifier made of: (1) a non-terminal from <strong>L</strong>, (2) an underscore (<strong>_</strong>), and (3) any other identifier. See the FAQ entry below.</p> + +<p>For example, <strong>(redex-match? L p t)</strong> returns <strong>#true</strong> if the term <strong>t</strong> matches the pattern <strong>p</strong> relative to the language <strong>L</strong>.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(redex-match? nat N_some-name (term Zero)) +;; #true +(redex-match? nat (Plus1 N_a) (term Zero)) +;; #false +(redex-match? nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; #true</code></pre> + +<p>If <strong>(redex-match? L p t)</strong> is <strong>#true</strong>, then <strong>(redex-match L p t)</strong> shows how named non-terminals in the pattern bind to subterms of <strong>t</strong>.</p> + +<pre><code>(redex-match nat N_0 (term Zero)) +;; (list (match (list (bind 'N_0 'Zero)))) +(redex-match nat (Plus1 N_0) (term Zero)) +;; #f +(redex-match nat (Plus1 N_0) (term (Plus1 (Plus1 Zero)))) +;; (list (match (list (bind 'N_0 '(Plus1 Zero)))))</code></pre> + +<h3 id="q-what-is-a-named-non-terminal">Q. What is a named non-terminal?</h3> + +<p>A named non-terminal in a language <strong>L</strong> is an identifier of the form <strong>X_Y</strong>, where:</p> + +<ul> + <li><strong>X</strong> is a non-terminal from <strong>L</strong></li> + <li><strong>Y</strong> is any identifier</li></ul> + +<p>The name helps when one pattern contains multiple occurrences of the same non-terminal. If you want the two occurrences to bind the same term, then use the same name.</p> + +<pre><code>(define-language trees + [binary-tree ::= Leaf + (Node binary-tree binary-tree)]) + +(redex-match trees + (Node binary-tree_left binary-tree_right) + (term (Node Leaf (Node Leaf Leaf)))) +;; (list +;; (match +;; (list (bind 'binary-tree_left 'Leaf) +;; (bind 'binary-tree_right '(Node Leaf Leaf)))))</code></pre> + +<h3 id="q-what-else-can-patterns-express">Q. What else can patterns express?</h3> + +<p>Redex patterns may contain special identifiers to guide pattern-matching. For instance:</p> + +<ul> + <li>The <strong>_</strong> pattern matches any term (and does not bind).</li> + <li>A pattern <strong>(p &hellip;)</strong> matches any sequence whose elements match the pattern <strong>p</strong>. If the pattern <strong>p</strong> is a named non-terminal, then the non-terminal binds a sequence of subterms.</li></ul> + +<p>Examples:</p> + +<pre><code>(redex-match? nat (Plus1 _) (term (Plus1 9))) +;; #true +(redex-match? nat (N_0 ...) (term ())) +;; #true +(redex-match? nat (N_0 ...) (term (Zero))) +;; #true +(redex-match nat (N_0 ...) (term (Zero Zero Zero))) +;; (list (match (list (bind 'N_0 '(Zero Zero Zero)))))</code></pre> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29">the Redex reference</a> for the full pattern language, including the <em>named and unique non-terminals</em> of the form <strong>X_!_Y</strong>.</p> + +<h3 id="q-what-can-patterns-not-express">Q. What can patterns <strong>not</strong> express?</h3> + +<ul> + <li>Disjunctions of patterns, e.g., &ldquo;number or boolean&rdquo;. (Use a language non-terminal.)</li> + <li>Negations of patterns. (Compose <strong>not</strong> with <strong>redex-match?</strong>.)</li> + <li>Some non-regular patterns. (Named dots <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._pattern%29"><code>..._N</code></a> or <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-language%29%29"><code>define-language</code></a> with a side condition may be able to help.)</li></ul> + +<h3 id="q-what-is-a-judgment">Q. What is a judgment?</h3> + +<p>A Redex <em>judgment</em> form defines a relation on terms. The relation is defined by a set of inference rules.</p> + +<p>Programming languages papers use inference rules all the time. Redex can express many of the judgments in papers; for example:</p> + +<ul> + <li>well-formedness conditions (i.e., whether a term contains free variables)</li> + <li>type checking rules</li> + <li>type inference rules</li> + <li>evaluation relations</li></ul> + +<p>Every judgment needs (1) a language (2) a mode (3) a contract (4) a set of inference rules. For example, the following judgment defines an equality relation on natural numbers.</p> + +<pre><code>(define-language nat + [N ::= Zero (Plus1 N)]) + +(define-judgment-form nat + #:mode (N= I I) + #:contract (N= N N) + [ + --- Zero= + (N= Zero Zero)] + [ + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1) + (N= N_0-- N_1--) + --- Plus1= + (N= N_0 N_1)])</code></pre> + +<ol> + <li>the language is <strong>nat</strong>; Redex uses the language to interpret patterns</li> + <li>the mode is <strong>(N= I I)</strong>; this means <strong>N=</strong> is the name of a judgment that expects two input terms (or, <strong>N=</strong> is a binary relation on terms)</li> + <li>the contract is <strong>(N= N N)</strong>; in other words, <strong>N=</strong> expects two terms that match the <strong>N</strong> non-terminal from the <strong>nat</strong> language</li> + <li>there are two inference rules, named <strong>Zero=</strong> and <strong>Plus1=</strong></li> + <li>the <strong>Zero=</strong> rule states that <strong>(N= Zero Zero)</strong> always holds</li> + <li>the <strong>Plus1=</strong> rule states that <strong>(N= N_0 N_1)</strong> holds if <strong>N_0</strong> and <strong>N_1</strong> are both <strong>Plus1</strong> terms whose contents are related by <strong>N=</strong></li></ol> + +<p>The <strong>where</strong> clauses are <em>guards</em>. When Redex tries to apply a rule with premises of the form <strong>(where pattern term)</strong>, it checks that each pattern matches the corresponding term. If not, Redex stops applying the rule. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-judgment-form%29%29">the Redex reference</a> for more.</p> + +<pre><code>(judgment-holds (N= Zero Zero)) +;; #true +(judgment-holds (N= (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(judgment-holds (N= (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Note: the inference rules form a <em>set</em>, not a <em>sequence</em>. So when you ask Redex whether <strong>(judgment-holds (N= Zero Zero))</strong>, it applies all rules that match <strong>(N= Zero Zero)</strong>. For <strong>N=</strong> this is just one rule, but in general it could be many rules.</p> + +<h3 id="q-what-is-a-judgment-form-mode">Q. What is a judgment form <strong>#:mode</strong>?</h3> + +<p>A <strong>#:mode</strong> declaration expects a list of the form <strong>(id pos-use &hellip;)</strong>, where <strong>id</strong> is an identifier and each <strong>pos-use</strong> is either <strong>I</strong> or <strong>O</strong>. These declarations say four things:</p> + +<ol> + <li><strong>id</strong> is the name of a new judgment form</li> + <li><strong>id</strong> expects <strong>N</strong> arguments, where <strong>N</strong> is the number of <strong>pos-use</strong> symbols</li> + <li><strong>id</strong> expects an <em>input</em> at each position where the mode contains an <strong>I</strong></li> + <li><strong>id</strong> produces an <em>output</em> at each position where the mode contains an <strong>O</strong></li></ol> + +<p>For example, a type inference judgment may take an expression as input and output a type. Here&rsquo;s a fast and easy type inference judgment for arithmetic expressions. Given any term <strong>e_0</strong>, the judgment outputs the type <strong>Int</strong>.</p> + +<pre><code>(define-language Arith + (e ::= integer (e + e)) + (τ ::= Int)) + +(define-judgment-form Arith + #:mode (infer-type I O) + #:contract (infer-type e τ) + [ + --- T-Int + (infer-type e_0 Int)])</code></pre> + +<h3 id="q-what-can-judgments-not-express">Q. What can judgments <strong>not</strong> express?</h3> + +<p>Redex does not support inference rules that require guessing.</p> + +<p>One example of this is a transitivity rule: "<strong>τ_0</strong> is related to <strong>τ_2</strong> if there exists a <strong>τ_1</strong> such that <strong>τ_0</strong> is related to <strong>τ_1</strong> and <strong>τ_1</strong> is related to <strong>τ_2</strong>". The following example tries to define a transitive subtyping (<strong>&lt;:</strong>) relation, but Redex rejects the <strong>S-Trans</strong> rule.</p> + +<pre><code>(define-language SomeTypes + (τ ::= (→ τ τ) Integer)) + +(define-judgment-form SomeTypes + #:mode (&lt;: I I) + #:contract (&lt;: τ τ) + [ + (&lt;: τ_0 τ_1) + (&lt;: τ_1 τ_2) + --- S-Trans + (&lt;: τ_0 τ_2)] + [ + --- S-Refl + (&lt;: τ_0 τ_0)] + [ + (&lt;: τ_dom-1 τ_dom-0) + (&lt;: τ_cod-0 τ_cod-1) + --- S-Arrow + (&lt;: (→ τ_dom-0 τ_cod-0) (→ τ_dom-1 τ_cod-1))])</code></pre> + +<p>The error is that in the rule <strong>S-Trans</strong>, the named non-terminal <strong>τ_1</strong> appears in an input position but is not bound to a term.</p> + +<h3 id="q-what-is-a-metafunction">Q. What is a metafunction?</h3> + +<p>A metafunction is a term-level function on terms.</p> + +<p>Every metafunction needs: (1) a language (2) a name (3) a contract (4) a sequence of guarded input/output cases.</p> + +<p>Here is a metafunction that returns <strong>#true</strong> when given two equal natural numbers. The definition is similar to the <strong>N=</strong> judgment form.</p> + +<pre><code>(define-metafunction nat + N=? : N N -&gt; boolean + [(N=? Zero Zero) + #true] + [(N=? N_0 N_1) + (N=? N_0-- N_1--) + (where (Plus1 N_0--) N_0) + (where (Plus1 N_1--) N_1)] + [(N=? N_0 N_1) + #false])</code></pre> + +<ul> + <li>the metafunction is named <strong>N=?</strong></li> + <li>its contract is <strong>N N -&gt; boolean</strong>, this means <strong>N=?</strong> expects 2 terms that match the <strong>N</strong> pattern and returns a term that matches the pattern <strong>boolean</strong></li> + <li>there are three cases; the second case is guarded by two <strong>where</strong> clauses</li></ul> + +<p>Any occurrence of <strong>(N=? &hellip;.)</strong> in any term is evaluated.</p> + +<pre><code>(term (N=? (Plus1 (Plus1 Zero)) (Plus1 (Plus1 Zero)))) +;; #true +(term ((N=? Zero Zero) Zero)) +;; '(#true Zero) +(term (N=? (Plus1 Zero) (Plus1 (Plus1 Zero)))) +;; #false</code></pre> + +<p>Any occurrence of <strong>N=?</strong> outside a <strong>term</strong> is an error.</p> + +<p>Metafunction <strong>where</strong>-clauses are analogous to judgment form <strong>where</strong>-clauses. See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28tech._metafunction%29">the Redex reference</a> for more.</p> + +<p>Note: the cases in a metafunction form a <em>sequence</em>, not a <em>set</em>. To evaluate a metafunction application, Redex tries each case in order and returns the result of the first case that (1) matches the call-site (2) for which all guards succeed.</p> + +<h3 id="q-should-i-use-a-metafunction-or-a-judgment-form">Q. Should I use a metafunction or a judgment form?</h3> + +<p>Use a judgment form.</p> + +<p>Metafunctions are handy, but judgments are easier to read and debug and maintain.</p> + +<h3 id="q-what-is-a-reduction-relation">Q. What is a reduction relation?</h3> + +<p>A reduction relation is a set of term-rewriting rules.</p> + +<p>Every reduction relation needs: (1) a language (2) a domain (3) a set of rewrite rules.</p> + +<ul> + <li>The language tells Redex how to interpret patterns.</li> + <li>The domain is a pattern. Input to the reduction relation must match the pattern, and output from the reduction relation must match the pattern.</li> + <li>The rewrite rules have the form <strong>(&mdash;&gt; term term guard &hellip;)</strong>. The term on the left is the input, the term on the right is the output.</li></ul> + +<p>See <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._reduction-relation%29%29">the Redex reference</a> for a full description of the guards.</p> + +<p>The preferred way to define a reduction relation is to define a language that includes three non-terminals:</p> + +<ol> + <li>a non-terminal for the domain of the reduction relation</li> + <li>a non-terminal for a <em>subset</em> of the domain that cannot reduce further</li> + <li>a non-terminal for evaluation contexts</li></ol> + +<p>An evaluation context is a term that contains a <strong>hole</strong>. A reduction relation can match a term against an evaluation context to find a sub-term to rewrite &mdash; in particular, the sub-term that matches the <strong>hole</strong>.</p> + +<p>In the following example, <strong>bexp</strong> is the domain of a reduction relation. A <strong>bexp</strong> term represents a boolean expression, which can be <strong>#true</strong> or <strong>#false</strong> or a conjunction of expressions or a disjunction of expressions. The boolean expressions <strong>#true</strong> and <strong>#false</strong> are also values (<strong>val</strong>); these cannot reduce further. The non-terminal <strong>E</strong> is for evaluation contexts.</p> + +<pre><code>(define-language Bool + (bexp ::= #true #false (bexp ∧ bexp) (bexp ∨ bexp)) + (val ::= #true #false) + (E ::= hole (E ∧ bexp) (val ∧ E) (E ∨ bexp) (val ∨ E))) + +(define step + (reduction-relation Bool + #:domain bexp + [--&gt; (in-hole E (val_lhs ∧ val_rhs)) + (in-hole E val_new) + ∧-step + (where val_new ,(and (term val_lhs) (term val_rhs)))] + [--&gt; (in-hole E (val_lhs ∨ val_rhs)) + (in-hole E val_new) + ∨-step + (where val_new ,(or (term val_lhs) (term val_rhs)))]))</code></pre> + +<p>The function <strong>apply-reduction-relation</strong> applies a reduction relation to a term and returns a list of ways that the term can step.</p> + +<pre><code>(apply-reduction-relation step (term #true)) +;; '() +(apply-reduction-relation step (term (#true ∧ #true))) +;; '(#true) +(apply-reduction-relation step (term (#true ∧ #false))) +;; '(#false) +(apply-reduction-relation step (term ((#true ∨ #false) ∧ #true))) +;; '((#true ∧ #true))</code></pre> + +<p>Three things about the reduction relation <strong>step</strong>:</p> + +<ol> + <li>Using <strong>in-hole</strong> on the first argument of <strong>&mdash;&gt;</strong> searches a term for a subterm that Redex can apply a reduction rule to.</li> + <li>Using <strong>in-hole</strong> on the second argument of <strong>&mdash;&gt;</strong> puts a new value back into the <strong>hole</strong> in the evaluation context.</li> + <li>The unquote operator (<strong>,</strong>) escapes to &ldquo;Racket mode&rdquo; (see below) to evaluate a conjunction or disjunction.</li></ol> + +<p>A judgment or metafunction is a formal alternative to &ldquo;escaping to Racket&rdquo;, but escaping can be convenient.</p> + +<p>Note: the cases in a reduction relation form a <em>set</em>, not a <em>sequence</em>. If more than one case matches, Redex applies them all.</p> + +<h3 id="q-what-is-racket-mode-what-is-redex-mode">Q. What is &ldquo;Racket mode&rdquo;? What is &ldquo;Redex mode&rdquo;?</h3> + +<p>Code in a Redex model is sometimes evaluated in &ldquo;Racket mode&rdquo; and sometimes evaluated in &ldquo;Redex mode&rdquo;. Racket mode evaluates Racket syntax to Racket values. Redex mode evaluates Racket syntax (possibly containing metafunction names) to terms.</p> + +<p>Key points:</p> + +<ol> + <li>A Redex program starts in Racket mode.</li> + <li>The <strong>X</strong> in <strong>(term X)</strong> is evaluated in Redex mode &hellip;</li> + <li>&hellip; unless <strong>X</strong> contains unquoted sub-expressions. Unquoting escapes to Racket mode &hellip;</li> + <li>&hellip; and <strong>term</strong>s inside an unquoted sub-expression are evaluated in Redex mode.</li></ol> + +<p>In other words, <strong>term</strong> enters Redex mode and <strong>unquote</strong> (<strong>,</strong>) escapes back to Racket.</p> + +<p>Redex implicitly switches to Redex mode in a few other places, for example:</p> + +<ul> + <li>the right-side of a <strong>where</strong> clause is in Redex mode</li> + <li>the result of a metafunction is in Redex mode</li></ul> + +<p>When in doubt, try using an <strong>unquote</strong>. Redex will raise an exception if it finds an unquote in Racket mode.</p> + +<h3 id="q-are-side-conditions-evaluated-in-racket-mode-or-redex-mode">Q. Are <strong>side-condition</strong>s evaluated in &ldquo;Racket mode&rdquo; or &ldquo;Redex mode&rdquo;?</h3> + +<p>A <strong>(side-condition e)</strong> sometimes evaluates <strong>e</strong> as a Racket expression and sometimes evaluates <strong>e</strong> as a Redex expression.</p> + +<ul> + <li>reduction relations and metafunctions expect a <strong>Racket</strong> expression</li> + <li>judgments expect a <strong>Redex</strong> expression</li></ul> + +<h3 id="q-what-is-a-binding-form">Q. What is a binding form?</h3> + +<p>In the lambda calculus, <strong>λ</strong>-terms bind variables. A term <strong>(λ x M)</strong> means that any free occurrence of <strong>x</strong> in the sub-term <strong>M</strong> refers to the <strong>x</strong> from the <strong>λ</strong>-term.</p> + +<p>Redex can express this idea with a binding form.</p> + +<pre><code>(define-language Λ + [e ::= (e e) x (λ x e)] + [x ::= variable-not-otherwise-mentioned] + #:binding-forms + (λ x_0 e_0 #:refers-to x_0))</code></pre> + +<p>Note: all the non-terminals in a language must be defined before the <strong>#:binding-forms</strong> keyword. If a non-terminal definition appears after the <strong>#:binding-forms</strong> keyword, then Redex will interpret the &ldquo;definition&rdquo; as a binding form.</p> + +<p>Binding forms work together with Redex&rsquo;s functions for substitution and alphabetic equivalence.</p> + +<pre><code>(alpha-equivalent? Λ + (term (λ x x)) + (term (λ y y)))) +;; #true + +(define-metafunction Λ + test-substitute : e -&gt; e + [(test-substitute (λ x_0 e_0)) + (substitute e_0 x_0 y)]) +(term (test-substitute (λ z (z z)))) +;; '(y y)</code></pre> + +<h3 id="q-what-is--what-is-">Q. What is <strong>&hellip;</strong>? What is <strong>&hellip;.</strong>?</h3> + +<p>Three dots (<strong>&hellip;</strong>) is for building patterns. If <strong>p</strong> is a pattern then <strong>(p &hellip;)</strong> matches any list whose elements all match <strong>p</strong>.</p> + +<pre><code>(define-language L) +(redex-match? L (number ... boolean ...) (term (1 2 #true #true))) +;; #true</code></pre> + +<p>Four dots (<strong>&hellip;.</strong>) may be used in <a href="http://docs.racket-lang.org/redex/The_Redex_Reference.html#%28form._%28%28lib._redex%2Freduction-semantics..rkt%29._define-extended-language%29%29"><strong>define-extended-language</strong></a> to extend a previously-defined non-terminal.</p> + +<pre><code>(define-language C + (keyword ::= auto break case)) +(define-extended-language C++ + C + (keyword ::= .... class)) + +(redex-match? C keyword (term auto)) +;; #true +(redex-match? C keyword (term class)) +;; #false +(redex-match? C++ keyword (term auto)) +;; #true +(redex-match? C++ keyword (term class)) +;; #true</code></pre> + +<h3 id="q-where-to-learn-more-about-redex">Q. Where to learn more about Redex?</h3> + +<p>&ldquo;Critical path&rdquo; resources:</p> + +<ul> + <li>Code examples for this post: <a href="https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt">https://github.com/nuprl/website/blob/master/blog/static/redex-faq.rkt</a></li> + <li>Redex documentation: <a href="http://docs.racket-lang.org/redex/index.html">http://docs.racket-lang.org/redex/index.html</a></li> + <li><a href="https://dvanhorn.github.io/redex-aam-tutorial/"><em>An Introduction to Redex with Abstracting Abstract Machines</em></a> by David Van Horn</li> + <li><a href="https://www.leafac.com/playing-the-game-with-plt-redex/"><em>Playing the Game with PLT Redex</em></a> by Leandro Facchinetti</li> + <li><a href="https://williamjbowman.com/doc/experimenting-with-redex/index.html"><em>Experimenting with Languages in Redex</em></a> by William J. Bowman</li></ul> + +<p>&ldquo;Procrastination&rdquo; resources:</p> + +<ul> + <li><a href="http://tata.gforge.inria.fr/"><em>Tree Automata Techniques and Applications</em></a></li> + <li><a href="http://lamport.azurewebsites.net/pubs/lamport-types.pdf"><em>Should your Specification Language be Typed?</em></a></li> + <li>Redex source code (see <code>redex-lib/</code>): <a href="https://github.com/racket/redex">https://github.com/racket/redex</a></li></ul> + + Building a Website with Scribble + http://prl.ccs.neu.edu/blog/2017/05/23/building-a-website-with-scribble/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-23-building-a-website-with-scribble + Tue, 23 May 2017 01:53:13 UT + Ben Greenman + +<p>The source code for the PRL website is written using Scribble, the Racket documentation tool. I am very happy with this choice, and you should be too!</p> +<!-- more--> + +<h2 id="the-story-so-far">The Story so Far</h2> + +<p>Last Fall, I took a flight to Chicago (on my way to <a href="http://con.racket-lang.org/2016/">RacketCon 2016</a>). When I landed, there was a new message in my inbox:</p> + +<pre><code> Subject: Web Page + Date: 2016-09-15 + + You have been nominated webmaster by public acclamation. Congratulations!</code></pre> + +<p>Emboldened by the trust of my people, I promptly converted the PRL website from Racket-generating-HTML to the fine <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> preprocessor language (commit <a href="https://github.com/nuprl/website/commit/a0600d32fec4bd70c5530b2717aec32979d634f7"><code>a0600d</code></a>) This bold action polarized the community.</p> + +<blockquote> + <p>I can&rsquo;t read the source anymore! Is this really an improvement?</p></blockquote> + +<p>Fear not, citizens. The switch to <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> was the right choice, and you too can learn to read the source code.</p> + +<h2 id="how-to-read-scribblehtml-programs">How to Read <code>scribble/html</code> Programs</h2> + +<h3 id="basics">Basics</h3> + +<p>Scribble is a language for writing Racket documentation. The key innovation in Scribble is the <em>@-expression</em> (read: &ldquo;at expression&rdquo;). The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language combines @-expression syntax with functions that generate HTML.</p> + +<h4 id="-syntax">@-syntax</h4> + +<p><a href="http://www.greghendershott.com/2015/08/at-expressions.html">Greg Hendershott</a> and the <a href="http://docs.racket-lang.org/scribble/reader.html">Scribble Documentation</a> explain @-expressions properly. Here&rsquo;s a short tutorial (Part 1 of 2, &ldquo;the basics&rdquo;):</p> + +<ul> + <li>Scribble programs start in &ldquo;text mode&rdquo;. Every character you type goes straight to the document you are building.</li> + <li>The @-sign toggles to &ldquo;Racket mode&rdquo; for the next expression. In Racket mode, the characters you type will be evaluated as a Racket program to produce part of the document.</li></ul> + +<p><em>Examples:</em> Evaluating <code>"Hello Dave"</code> puts &ldquo;Hello Dave&rdquo; in your document. Evaluating <code>"Hello @Dave"</code> puts &ldquo;Hello ???&rdquo; in your document, where "???" is the value of the variable <code>Dave</code>. Finally if <code>Dave</code> is the name of a function, then <code>"Hello @(Dave)"</code> calls the <code>Dave</code> function with zero arguments and puts whatever it returns into your document.</p> + +<p>To make it easy to interleave text, function calls, and code, Scribble discriminates between 4 kinds of parentheses when they follow an @-sign (Part 2 of 2, &ldquo;the parens&rdquo;):</p> + +<ul> + <li><code>@(f A B)</code> is just like the function call <code>(f A B)</code> in Racket</li> + <li><code>@f[A B]</code> is the same as <code>@(f A B)</code>, but typically more useful because &hellip;</li> + <li><code>@f[A B]{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; to a list of words <code>w*</code>, then calls <code>f</code> just like <code>(apply f A B w*)</code></li> + <li><code>@f{....}</code> evaluates the <code>....</code> in &ldquo;text mode&rdquo; and calls <code>f</code> with the results</li> + <li><code>@f|{....}|</code> is similar, but the <code>....</code> are in &ldquo;unescapable text mode&rdquo;</li></ul> + +<p>&ldquo;Unescapable text mode&rdquo; treats @-signs as text instead of toggling between modes.</p> + +<h4 id="generating-html">Generating HTML</h4> + +<p>The <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a> language comes with functions that render HTML. These functions have the same name as the corresponding HTML tag.</p> + +<p>Example program:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Running this program prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>No surprises.</p> + +<p>One thing that <em>is</em> surprising is how <code>scribble/html</code> handles tag attributes. Every tag-rendering function accepts &ldquo;Racket mode&rdquo; arguments that specify an attribute name and attribute value.</p> + +<p>For example:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@p</span><span class="p">[</span><span class="n">style:</span> <span class="s2">"color:red"</span><span class="p">]{</span><span class="n">Hello</span> <span class="n">World</span><span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Prints:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">&gt;</span>Hello World<span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Hope the output looks familiar. The input syntax is strange, but that&rsquo;s what it is.</p> + +<p>Larger programs print larger webpages. Each page on the PRL website is HTML generated by one <code>scribble/html</code> program.</p> + +<h2 id="why-scribblehtml-is-an-improvement">Why <code>scribble/html</code> is an Improvement</h2> + +<p>Before <code>scribble/html</code>, the PRL website was implemented in <code>scribble/text</code>. A <code>scribble/text</code> program renders and prints text. There is no extra support for HTML.</p> + +<p>To compare, here&rsquo;s the start of the old homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/text</span> +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._require))" style="color: inherit">require</a></span> <span class="s2">"templates.rkt"</span><span class="p">)</span> + +<span class="n">&lt;!DOCTYPE</span> <span class="n">html&gt;</span> +<span class="n">&lt;html</span> <span class="n">lang=</span><span class="s2">"en"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">header</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;body</span> <span class="n">id=</span><span class="s2">"pn-top"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> + <span class="n">@</span><span class="p">(</span><span class="n">navbar</span> <span class="s2">"Home"</span><span class="p">)</span> + <span class="n">&lt;div</span> <span class="n">class=</span><span class="s2">"jumbotron"</span><span class="nb"><a href="http://docs.racket-lang.org/reference/generic-numbers.html#(def._((quote._~23~25kernel)._~3e))" style="color: inherit">&gt;</a></span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And here is the start of the <code>scribble/html</code>&rsquo;d homepage:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span> +<span class="normal">9</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="kn">#lang </span><span class="nn">scribble/html</span> +<span class="n">@require</span><span class="p">[</span><span class="s2">"templates.rkt"</span><span class="p">]</span> + +<span class="n">@doctype</span><span class="p">{</span><span class="n">html</span><span class="p">}</span> +<span class="n">@html</span><span class="p">[</span><span class="n">lang:</span> <span class="s2">"en"</span><span class="p">]{</span> + <span class="n">@header</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@body</span><span class="p">[</span><span class="n">id:</span> <span class="s2">"pn-top"</span><span class="p">]{</span> + <span class="n">@navbar</span><span class="p">{</span><span class="n">Home</span><span class="p">}</span> + <span class="n">@div</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"jumbotron"</span><span class="p">]{</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The pages look similar. The new one has more @-signs and parentheses, the old one has more <code>&lt;</code>-signs and quotes. If you were able to edit the old page, you should be able to edit the new page.</p> + +<p>The <strong>key improvement</strong> in the new page is that <strong>common mistakes are now compile-time errors</strong>.</p> + +<ul> + <li> + <p>Before, a typo like <code>&lt;hmtl&gt;</code> would generate an ugly webpage. After, a typo like <code>@hmtl</code> is a syntax error.</p></li> + <li> + <p>Before, a typo like <code>&lt;b&gt;....</code> with no closing tag would generate an ugly webpage. After, a typo like <code>@b{....</code> is a syntax error.</p></li></ul> + +<p>Both flavors of error message come with source-code line numbers. This is very very helpful.</p> + +<h3 id="small-improvements">Small Improvements</h3> + +<h4 id="1-more-functions">1. More Functions</h4> + +<p>Before, the <a href="http://prl.ccs.neu.edu/teaching.html">Teaching page</a> contained some interesting HTML for rendering vertical text (look for the word &ldquo;Semantics&rdquo; to see how this was used):</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"how-to-design-programs"</span><span class="p">&gt;</span>S<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>e<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>m<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>a<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>n<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>t<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>i<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>c<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span>s<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;&lt;/</span><span class="nt">span</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>After, the same text is generated from a function call:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@span</span><span class="p">[</span><span class="n">class:</span> <span class="s2">"how-to-design-programs"</span><span class="p">]{</span><span class="n">@vertical-text</span><span class="p">{</span><span class="n">Semantics</span><span class="p">}}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The <code>vertical-text</code> function is simple:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@require</span><span class="p">[(</span><span class="k"><a href="http://docs.racket-lang.org/reference/require.html#(form._((lib._racket/private/base..rkt)._only-in))" style="color: inherit">only-in</a></span> <span class="n">racket/list</span> <span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span><span class="p">)]</span> + +<span class="n">@</span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="n">vertical-text</span> <span class="o">.</span> <span class="n">str*</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._add-between))" style="color: inherit">add-between</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-~3elist))" style="color: inherit">string-&gt;list</a></span> <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/pairs.html#(def._((lib._racket/list..rkt)._append*))" style="color: inherit">append*</a></span> <span class="n">str*</span><span class="p">))</span> <span class="p">(</span><span class="n">br</span><span class="p">)))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="2-more-structure-less-boilerplate">2. More Structure, Less Boilerplate</h4> + +<p>Here&rsquo;s part of the old definition of &ldquo;Ben Greenman&rdquo; on the <a href="http://prl.ccs.neu.edu/people.html">People page</a>:</p> + +<div class="brush: html"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span> +<span class="normal">11</span> +<span class="normal">12</span> +<span class="normal">13</span> +<span class="normal">14</span> +<span class="normal">15</span> +<span class="normal">16</span> +<span class="normal">17</span> +<span class="normal">18</span> +<span class="normal">19</span> +<span class="normal">20</span> +<span class="normal">21</span> +<span class="normal">22</span> +<span class="normal">23</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"row pn-person"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-row-eq-height"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-photo"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"img-wrapper"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">img</span> <span class="na">src</span><span class="o">=</span><span class="s">"img/ben_greenman.jpg"</span> <span class="na">title</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="na">alt</span><span class="o">=</span><span class="s">"Ben Greenman"</span> <span class="p">/&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-9"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-4 pn-contact"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"pn-name"</span><span class="p">&gt;</span>Ben Greenman<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + Advisor: Matthias Felleisen<span class="p">&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"mailto:types@"</span><span class="err">@"</span><span class="na">ccs</span><span class="err">.</span><span class="na">neu</span><span class="err">.</span><span class="na">edu</span><span class="err">"</span><span class="p">&gt;</span>types@"@"ccs.neu.edu<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;&lt;</span><span class="nt">br</span> <span class="p">/&gt;</span> + <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"http://www.ccs.neu.edu/home/types"</span><span class="p">&gt;</span>www.ccs.neu.edu/home/types<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-3 pn-muted col-md-offset-5"</span><span class="p">&gt;</span> + Joined 2014 + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"col-md-12 pn-bio"</span><span class="p">&gt;</span> + <span class="p">&lt;</span><span class="nt">p</span><span class="p">&gt;</span>I like constructions .... <span class="p">&lt;/</span><span class="nt">p</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> + <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new definition uses a helper function with keyword arguments for each &ldquo;field&rdquo; of the person:</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span> +<span class="normal">6</span> +<span class="normal">7</span> +<span class="normal">8</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@person</span><span class="p">[</span><span class="kd">#:name</span> <span class="s2">"Ben Greenman"</span> + <span class="kd">#:title</span> <span class="s2">"Advisor: Matthias Felleisen"</span> + <span class="kd">#:e-mail</span> <span class="s2">"types@ccs.neu.edu"</span> + <span class="kd">#:website</span> <span class="s2">"http://ccs.neu.edu/home/types"</span> + <span class="kd">#:history</span> <span class="n">@list</span><span class="p">[</span><span class="s2">"Joined 2014"</span><span class="p">]</span> + <span class="kd">#:img</span> <span class="s2">"ben_greenman.jpg"</span><span class="p">]{</span> + <span class="n">I</span> <span class="n">like</span> <span class="n">constructions</span> <span class="n">....</span> +<span class="p">}</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="3-less-string-formatting">3. Less String-Formatting</h4> + +<p>Before, the code did a lot of string formatting (<a href="https://github.com/nuprl/website/commit/a0600d#diff-1921e33ce89be28dd277cf1c7880d1beL9">link</a>):</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/define.html#(form._((lib._racket/private/base..rkt)._define))" style="color: inherit">define</a></span> <span class="p">(</span><span class="k"><a href="http://docs.racket-lang.org/reference/creatingunits.html#(form._((lib._racket/unit..rkt)._link))" style="color: inherit">link</a></span> <span class="n">url</span> <span class="n">body</span><span class="p">)</span> + <span class="p">(</span><span class="nb"><a href="http://docs.racket-lang.org/reference/strings.html#(def._((quote._~23~25kernel)._string-append))" style="color: inherit">string-append</a></span> <span class="s2">"&lt;a href=</span><span class="se">\"</span><span class="s2">"</span> <span class="n">url</span> <span class="s2">"</span><span class="se">\"</span><span class="s2">&gt;"</span> <span class="n">body</span> <span class="s2">"&lt;/a&gt;"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>The new code has no need for such helper functions.</p> + +<div class="brush: racket"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="n">@a</span><span class="p">[</span><span class="n">href:</span> <span class="n">url</span> <span class="n">body</span><span class="p">]</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h4 id="bottom-line">Bottom Line</h4> + +<p>Scribble is a good language for making static HTML pages.</p> + +<hr /> + +<p><em>If you liked this post, you may also be interested in:</em></p> + +<ul> + <li><a href="http://docs.racket-lang.org/pollen/index.html">Pollen</a></li> + <li><a href="https://github.com/vishesh/racketscript">RacketScript</a></li> + <li>Other websites built using <a href="http://docs.racket-lang.org/scribble-pp/html.html"><code>scribble/html</code></a>: (1) <a href="http://nanopass.org/">nanopass.github.io</a> (<a href="https://github.com/nanopass/nanopass.github.io">source code</a>), (2) <a href="http://prl.ccs.neu.edu/gtp/">Gradual Typing Across the Spectrum</a> (<a href="https://github.com/nuprl/gtp">source code</a>).</li> + <li><a href="http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/">Notes from a Gradual Typing Across the Spectrum PI meeting</a></li></ul> + + [Getting Started in Programming Languages (Cross-Post)](http://jschuster.org/blog/2016/11/29/getting-started-in-programming-languages/) + http://prl.ccs.neu.edu/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-11-30-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages + Wed, 30 Nov 2016 15:24:45 UT + Jonathan Schuster + + + CompCert Overview + http://prl.ccs.neu.edu/blog/2016/10/11/compcert-overview/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-10-11-compcert-overview + Tue, 11 Oct 2016 17:41:16 UT + Ben Greenman + +<p>If you are interested in learning about the <em>internals</em> of the CompCert C compiler but would rather not read its source code, this post is for you.</p> +<!-- more--> + +<p>(This is a public service announcement.)</p> + +<p>Last fall, I gave a short lecture on the 2006 paper <a href="http://gallium.inria.fr/~xleroy/publi/compiler-certif.pdf">&ldquo;Formal Certification of a Compiler Back-End&rdquo;</a> by Xavier Leroy for Amal Ahmed&rsquo;s <a href="http://www.ccs.neu.edu/home/amal/course/7480-f15/">&ldquo;Special Topics in Programming Languages&rdquo;</a> class. Rather than present CompCert as it existed in 2006, I read the documentation and source code for <a href="https://github.com/AbsInt/CompCert/releases/tag/v2.5">CompCert 2.5</a> (released June 2015). The lecture then focused on three questions:</p> + +<ul> + <li>What subset of C does CompCert handle, today?</li> + <li>What optimizing passes does CompCert perform?</li> + <li>What is the &ldquo;correctness theorem&rdquo; for CompCert, and what does this theorem mean?</li></ul> + +<p>My notes for the lecture give a &ldquo;mid-level&rdquo; summary of the compiler &mdash; there are more details than you&rsquo;ll find in papers, but it&rsquo;s (hopefully!) easier to read than the source code. The document is also hyperlinked to locations in the <a href="https://github.com/AbsInt/CompCert">CompCert GitHub repository</a>.</p> + +<p>Here is the document:</p> + +<blockquote> + <p> <a href="http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf">http://www.ccs.neu.edu/home/types/resources/notes/compcert/cc.pdf</a></p></blockquote> + +<p>And here is a table-of-contents:</p> + +<ol> + <li>Motivation, details of the source and target languages, high-level guarantees</li> + <li>Compiler pipeline, optimizing passes, links intermediate language grammars and Coq theorems</li> + <li>Background on compiler correctness</li> + <li>CompCert&rsquo;s correctness, properties that CompCert does <strong>not</strong> guarantee</li> + <li>Recent (2006 &ndash; 2015) work in the CompCert ecosystem</li></ol> + +<p>The document ends with a short description of two other research projects that have grown into &ldquo;industry software&rdquo; and a link to Xaver Leroy&rsquo;s <a href="https://www.cs.uoregon.edu/research/summerschool/summer12/curriculum.html">OPLSS lectures on certified compilers</a>. Enjoy!</p> + + Tutorial: Zero to Sixty in Racket + http://prl.ccs.neu.edu/blog/2016/08/02/tutorial-zero-to-sixty-in-racket/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-08-02-tutorial-zero-to-sixty-in-racket + Tue, 02 Aug 2016 01:29:11 UT + Ben Greenman + +<p>Racket is excellent for incrementally growing scripts into full-fledged programs. +This post steps through the evolution of one small program and highlights the + Racket tools that enable incremental advances.</p> +<!--more--> + +<p></p> + +<div class="SIntrapara">Why should anyone use <a href="http://racket-lang.org/">Racket</a>? +There are two reasons: +</div> + +<div class="SIntrapara"> + <ol> + <li> + <p>You have a problem that can only be solved with Racket&rsquo;s language-building tools</p></li> + <li> + <p>Racket is a nice language to program in. +(Has lexical scope, parentheses, <a href="http://con.racket-lang.org">active users</a>...)</p></li></ol></div> + +<p>My favorite part of Racket is how it supports a certain development style of + evolving scripts into programs. +When I start coding (after design, before debugging), I can focus the problem at hand. +Next come examples, unit tests, and types to be sure the solution is correct. +Finally, I worry about aesthetics, efficiency, and how the solution can be used as a library in a larger context.</p> + +<p>Bottom line: with Racket, my coding is aligned with my priorities. +And as I transition from "no code" to "working code" to "robust code" to "re-usable code", + the program is almost always runnable.</p> + +<h1><a name="(part._.Problem__.A_.K.W.I.C_.Index_.Production_.System)"></a>Problem: A KWIC Index Production System</h1> + +<p>A KWIC index system +reads input from a file, +divides each line of the file into whitespace-separated words, +and outputs (in alphabetical order) all circular shifts of all lines.</p> + +<p>The first circular shift of a line <span class="RktWrap"><span class="RktVal">"A B C"</span></span> is the line <span class="RktWrap"><span class="RktVal">"B C A"</span></span>. +The second circular shift is <span class="RktWrap"><span class="RktVal">"C A B"</span></span>.</p> + +<p>Building a KWIC index is a historical problem. +According to <a href="https://www.cs.umd.edu/class/spring2003/cmsc838p/Design/criteria.pdf">D.L. Parnas (1972)</a>:</p> + +<blockquote class="SubFlow"> + <p>Except under extreme circumstances (huge data base, no supporting software) +such a system could be implemented by a good programmer within a week or two.</p></blockquote> + +<p>See also: <a href="https://yanniss.github.io/law.html">Yannis&rsquo;s Law</a>.</p> + +<p>Today, I bet only <a href="http://wiki.portal.chalmers.se/agda/pmwiki.php">Agda</a> and <a href="https://scratch.mit.edu/">Scratch</a> + programmers would need the full two weeks. +We&rsquo;ll be done in 20 minutes.</p> + +<h1><a name="(part._.A_.Script)"></a>A Script</h1> + +<p>To start, open a file and type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>You can name the file anything, like <span class="stt">kwic.rkt</span> or <span class="stt">rkt.kwic</span> or <span class="stt">foo</span>. +Racket doesn&rsquo;t care, + but it does need the <span class="stt">#lang</span> line to read the contents of the file.</p> + +<p>Though, you should use the <span class="stt">.rkt</span> extension.</p> + +<p>The first part of the solution is a function to read input from a file into a + list of strings for further processing. +The built-in function <a href="http://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28lib._racket%2Ffile..rkt%29._file-~3elines%29%29">file-&gt;lines</a> + does exactly this, but we&rsquo;ll for-loop instead.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="hspace">&nbsp;</span><span class="RktSym">filename</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="hspace">&nbsp;</span><span class="RktSym">filename</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>When called with a filename like <span class="RktWrap"><span class="RktVal">"heart-of-darkness.txt"</span></span>, the function + uses <a href="http://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a> to build a list of lines by reading from a port + with <a href="http://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a>. +The port is the data from <span class="RktWrap"><span class="RktSym">filename</span></span>, thanks to <a href="http://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a>.</p> + +<p>Next is a function to convert a list of strings into a list of lists of words. +Here we&rsquo;ll just use library functions.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktSym">lines</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="hspace">&nbsp;</span><span class="RktSym">string-split</span><span class="hspace">&nbsp;</span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>By default, <a href="http://docs.racket-lang.org/reference/strings.html#%28def._%28%28lib._racket%2Fstring..rkt%29._string-split%29%29">string-split</a> divides a string into a list of whitespace-separated substrings. +You can always supply a different delimiter, or use <a href="http://docs.racket-lang.org/reference/regexp.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._regexp-split%29%29">regexp-split</a> + to divide by a regular expression.</p> + +<p>Two tasks left! +First we generate all circular shifts for a list of strings <span class="RktWrap"><span class="RktSym">words</span></span> + by folding up a list with one shift of <span class="RktWrap"><span class="RktSym">words</span></span> for each word.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Second, we alphabetize and print the shifts.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">destruct multiple values</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">first list empty, don</span><span class="RktCmt">'</span><span class="RktCmt">t care about second</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#t</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">first list non-empty, second empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">s1</span><span class="hspace">&nbsp;</span><span class="RktSym">s2</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="hspace">&nbsp;</span><span class="RktSym">shift1-rest</span><span class="hspace">&nbsp;</span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="hspace">&nbsp;</span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktVal">" "</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="hspace">&nbsp;</span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">for-each is like map, but returns (void)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="hspace">&nbsp;</span><span class="RktSym">display-words</span><span class="hspace">&nbsp;</span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Gluing it all together, here&rsquo;s the full script (with type annotations in comments).</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">type</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">=</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">type</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">=</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Path-String</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">String)</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Move</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">element</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">last</span><span class="hspace">&nbsp;</span><span class="RktCmt">position</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lexicographic</span><span class="hspace">&nbsp;</span><span class="RktCmt">order</span><span class="hspace">&nbsp;</span><span class="RktCmt">on</span><span class="hspace">&nbsp;</span><span class="RktCmt">equal-length</span><span class="hspace">&nbsp;</span><span class="RktCmt">lists</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">Words</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Boolean</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">don't</span><span class="hspace">&nbsp;</span><span class="RktCmt">care</span><span class="hspace">&nbsp;</span><span class="RktCmt">about</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">non-empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/booleans.html#%28def._%28%28quote._~23~25kernel%29._not%29%29">not</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._null~3f%29%29">null?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"</span><span class="hspace">&nbsp;</span><span class="RktVal">"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">(Listof</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lines)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Path-String</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">append*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">End-to-end</span><span class="hspace">&nbsp;</span><span class="RktCmt">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-&gt;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">run-test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"test.txt"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Make</span><span class="hspace">&nbsp;</span><span class="RktCmt">a</span><span class="hspace">&nbsp;</span><span class="RktCmt">file</span><span class="hspace">&nbsp;</span><span class="RktCmt">and</span><span class="hspace">&nbsp;</span><span class="RktCmt">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/when_unless.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._unless%29%29">unless</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._file-exists~3f%29%29">file-exists?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test-file</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">run-test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Running the file should print:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">this</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">imagine</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._write%29%29">write</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">took</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">weeks</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">to</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<h1><a name="(part._.Testing_and_.Submodules)"></a>Testing and Submodules</h1> + +<p>Any top-level expressions in a file can work as unit tests. +The <a href="http://docs.racket-lang.org/reference/booleans.html%3F#%28def._%28%28quote._~23~25kernel%29._equal~3f%29%29">equal?</a> statement below checks whether the first circular shift + of <span class="RktWrap"><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="stt"> </span><span class="RktVal">"B"</span><span class="stt"> </span><span class="RktVal">"C"</span><span class="RktVal">)</span></span> is <span class="RktWrap"><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="stt"> </span><span class="RktVal">"C"</span><span class="stt"> </span><span class="RktVal">"A"</span><span class="RktVal">)</span></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Equality.html#%28def._%28%28quote._~23~25kernel%29._equal~3f%29%29">equal?</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Running the file now prints <span class="RktWrap"><span class="RktVal">#t</span></span> to the console, meaning the test passed. +We can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/exns.html#%28def._%28%28quote._~23~25kernel%29._error%29%29">error</a></span></span> or <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/exns.html#%28def._%28%28quote._~23~25kernel%29._raise-user-error%29%29">raise-user-error</a></span></span> to make failures easier + to notice. +Or we can use the <a href="http://docs.racket-lang.org/rackunit/api.html">RackUnit</a> testing library.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">import the testing library</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>These tests run each time the module does. +If you prefer to run tests only in a specific context, and not when the + module is run or imported as a library, you can move them to a separate + file or into a <a href="http://docs.racket-lang.org/reference/eval-model.html#%28tech._submodule%29">submodule</a>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Open a submodule named </span><span class="RktCmt">'</span><span class="RktCmt">test</span><span class="RktCmt">'</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Running the module normally via <span class="stt">racket kwic.rkt</span> will not run code + in the submodule. +Instead, use <span class="stt">raco test</span> to run the tests.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">raco</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">raco</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._submod%29%29">submod</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"kwic.rkt"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">passed</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The reason we used <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span></span>, instead of Racket&rsquo;s <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%29%29">module</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%2A%29%29">module*</a></span></span> + forms is that <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span></span> inherits the language and namespace of its + containing module and can be incrementally extended. +This way, we can keep tests near the relevant code.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">rackunit</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"A"</span><span class="hspace">&nbsp;</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"B"</span><span class="hspace">&nbsp;</span><span class="RktVal">"C"</span><span class="hspace">&nbsp;</span><span class="RktVal">"A"</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="hspace">&nbsp;</span><span class="RktSym">all-shifts</span><span class="hspace">&nbsp;</span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"is"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"as"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"does"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"as"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"does"</span><span class="RktVal">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"racket"</span><span class="hspace">&nbsp;</span><span class="RktVal">"is"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p><a href="http://docs.racket-lang.org/rackunit/api.html">RackUnit</a> in a + separate file or <span class="RktWrap"><span class="RktSym">test</span></span> submodule is the unofficial standard for testing + Racket programs.</p> + +<h1><a name="(part._.Recognizing_.Patterns__.Avoiding_.Repetition)"></a>Recognizing Patterns, Avoiding Repetition</h1> + +<p>Every unit test we&rsquo;ve written uses <a href="http://docs.racket-lang.org/rackunit/api.html#%28def._%28%28lib._rackunit%2Fmain..rkt%29._check-equal~3f%29%29">check-equal?</a>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"hello"</span><span class="hspace">&nbsp;</span><span class="RktVal">"world"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">" lost "</span><span class="hspace">&nbsp;</span><span class="RktVal">" in "</span><span class="hspace">&nbsp;</span><span class="RktVal">"space"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"lost"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"in"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"space"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"something"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>These tests follow a simple pattern that we can express as a <span class="emph">syntax rule</span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">test</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._define-syntax-rule%29%29">define-syntax-rule</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="hspace">&nbsp;</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="RktPn">]</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/begin.html#%28form._%28%28quote._~23~25kernel%29._begin%29%29">begin</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"hello"</span><span class="hspace">&nbsp;</span><span class="RktVal">"world"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">" out "</span><span class="hspace">&nbsp;</span><span class="RktVal">" in "</span><span class="hspace">&nbsp;</span><span class="RktVal">"the ozone"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">(</span><span class="RktVal">"out"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"in"</span><span class="RktVal">)</span><span class="hspace">&nbsp;</span><span class="RktVal">(</span><span class="RktVal">"the"</span><span class="hspace">&nbsp;</span><span class="RktVal">"ozone"</span><span class="RktVal">)</span><span class="RktVal">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"something"</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span></span> are not pseudocode! +They denote Kleene-star repetition, like a sextile (<span class="RktWrap"><span class="RktVal">"*"</span></span>) in a regular expression. +In this case, the input pattern is a sequence of lists with two S-expressions, <span class="RktWrap"><span class="RktSym">i</span></span> and + <span class="RktWrap"><span class="RktSym">o</span></span>. +Uses of <span class="RktWrap"><span class="RktSym">i</span></span> and <span class="RktWrap"><span class="RktSym">o</span></span> in the rule must be followed by one <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span></span> to splice + the captured S-expressions into the result.</p> + +<p>Many languages offer higher-order functions and polymorphism to abstract common + behaviors. +Syntax extensions are a different way to avoid repeating yourself. +After 30 years, we are still discovering what syntax extensions are useful for.</p> + +<p>See this <a href="https://groups.google.com/forum/#!topic/racket-users/ss20lwfUhjs/discussion">recent Racket mailing list post</a> for some applications.</p> + +<h1><a name="(part._.Adding_.Static_.Types)"></a>Adding Static Types</h1> + +<p>Changing the <a href="http://docs.racket-lang.org/reference/reader.html#%28idx._%28gentag._79._%28lib._scribblings%2Freference%2Freference..scrbl%29%29%29"><span class="stt">#lang</span></a> line to <span class="RktWrap"><span class="RktSym">typed/racket</span></span> adds static type-checking to our program. +If we only change the language and run the code as-is, there will be type errors. +But we can use submodules again to incrementally check our design with types.</p> + +<p>Note: <a href="http://docs.racket-lang.org/ts-reference/Typed_Regions.html">typed regions</a> + are another way to embed typed code into untyped contexts.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28quote._~23~25kernel%29._module%29%29">module</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">typed/racket</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Need</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">annotate:</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-</span><span class="hspace">&nbsp;</span><span class="RktCmt">function</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameters</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">-</span><span class="hspace">&nbsp;</span><span class="RktCmt">for-loop</span><span class="hspace">&nbsp;</span><span class="RktCmt">return</span><span class="hspace">&nbsp;</span><span class="RktCmt">types</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Next</span><span class="hspace">&nbsp;</span><span class="RktCmt">migration</span><span class="hspace">&nbsp;</span><span class="RktCmt">step:</span><span class="hspace">&nbsp;</span><span class="RktCmt">move</span><span class="hspace">&nbsp;</span><span class="RktCmt">other</span><span class="hspace">&nbsp;</span><span class="RktCmt">untyped</span><span class="hspace">&nbsp;</span><span class="RktCmt">functions</span><span class="hspace">&nbsp;</span><span class="RktCmt">here</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._all-defined-out%29%29">all-defined-out</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">t</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">&lt;rest</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">file</span><span class="hspace">&nbsp;</span><span class="RktCmt">omitted&gt;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>After scooping all functions into the Typed Racket bubble, we can remove the + submodule declaration and change <span class="stt">#lang racket</span> to <span class="stt">#lang typed/racket</span>.</p> + +<h1><a name="(part._.Finally__a_.Library)"></a>Finally, a Library</h1> + +<p>Other modules can import our functions if we use a <a href="http://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a> statement. +By <a href="https://docs.racket-lang.org/style/Units_of_Code.html">convention</a>, exports belong at the top of a file.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">#lang</span><span class="hspace">&nbsp;</span><span class="RktMeta">typed/racket</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._provide%29%29">provide</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-index</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">&lt;definitions</span><span class="hspace">&nbsp;</span><span class="RktCmt">here&gt;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Then any typed or untyped module can use <span class="RktWrap"><span class="RktSym">kwic-index</span></span> by writing + <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="stt"> </span><span class="RktVal">"kwic.rkt"</span><span class="RktPn">)</span></span>.</p> + +<p>As a finishing touch, we can use the <a href="http://docs.racket-lang.org/reference/Command-Line_Parsing.html">racket/cmdline</a> library + inside a <span class="stt">main</span> submodule to give a basic front-end interface. +Similar to <span class="stt">module+ test</span>, a <span class="stt">module+ main</span> declares code that + inherits the file&rsquo;s bindings and language but is only run when the program + is executaed.</p> + +<p>Here is the complete typed and tested code listing. +The <span class="RktWrap"><span class="RktSym">main</span></span> submodule is at the bottom.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/ts-reference/index.html"><span class="RktSym">typed/racket</span></a><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">typed/rackunit</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._define-syntax-rule%29%29">define-syntax-rule</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">o</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/begin.html#%28form._%28%28quote._~23~25kernel%29._begin%29%29">begin</a></span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">o</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._......%29%29">...</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">define-type</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">define-type</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-input-from-file%29%29">with-input-from-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">filename</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">line</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-lines%29%29">in-lines</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">make-temporary-file</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"The</span><span class="hspace">&nbsp;</span><span class="RktVal">Nellie,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"a</span><span class="hspace">&nbsp;</span><span class="RktVal">cruising</span><span class="hspace">&nbsp;</span><span class="RktVal">yawl,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"swung</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">her</span><span class="hspace">&nbsp;</span><span class="RktVal">anchor</span><span class="hspace">&nbsp;</span><span class="RktVal">without</span><span class="hspace">&nbsp;</span><span class="RktVal">a</span><span class="hspace">&nbsp;</span><span class="RktVal">flutter</span><span class="hspace">&nbsp;</span><span class="RktVal">of</span><span class="hspace">&nbsp;</span><span class="RktVal">sails,"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"and</span><span class="hspace">&nbsp;</span><span class="RktVal">was</span><span class="hspace">&nbsp;</span><span class="RktVal">at</span><span class="hspace">&nbsp;</span><span class="RktVal">rest."</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">actual</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expect</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">file-&gt;lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._delete-file%29%29">delete-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">actual</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expect</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">String</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#{</span><span class="RktSym">string-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta">::</span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"hello</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">world"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"hello"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"world"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Move</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">element</span><span class="hspace">&nbsp;</span><span class="RktCmt">to</span><span class="hspace">&nbsp;</span><span class="RktCmt">last</span><span class="hspace">&nbsp;</span><span class="RktCmt">position</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._append%29%29">append</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Ffold%29%29">for/fold</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">circular-shift</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Flist..rkt%29._sort%29%29">sort</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">Lexicographic</span><span class="hspace">&nbsp;</span><span class="RktCmt">order</span><span class="hspace">&nbsp;</span><span class="RktCmt">on</span><span class="hspace">&nbsp;</span><span class="RktCmt">equal-length</span><span class="hspace">&nbsp;</span><span class="RktCmt">lists</span><span class="hspace">&nbsp;</span><span class="RktCmt">of</span><span class="hspace">&nbsp;</span><span class="RktCmt">words</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Boolean</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">match*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">don't</span><span class="hspace">&nbsp;</span><span class="RktCmt">care</span><span class="hspace">&nbsp;</span><span class="RktCmt">about</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">first</span><span class="hspace">&nbsp;</span><span class="RktCmt">list</span><span class="hspace">&nbsp;</span><span class="RktCmt">non-empty,</span><span class="hspace">&nbsp;</span><span class="RktCmt">second</span><span class="hspace">&nbsp;</span><span class="RktCmt">empty</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._or%29%29">or</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3c~3f%29%29">string&lt;?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._and%29%29">and</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3d~3f%29%29">string=?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">s2</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift1-rest</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">shift2-rest</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktSym">shift&lt;?</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#t</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">first</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">word</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-list%29%29">in-list</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cdr%29%29">cdr</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">words</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"</span><span class="hspace">&nbsp;</span><span class="RktVal">"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28quote._~23~25kernel%29._display%29%29">display</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">word</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Byte_and_String_Output.html#%28def._%28%28quote._~23~25kernel%29._newline%29%29">newline</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._for-each%29%29">for-each</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">display-words</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-sorted-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._open-output-string%29%29">open-output-string</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._get-output-string%29%29">get-output-string</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A\nB</span><span class="hspace">&nbsp;</span><span class="RktVal">C\n"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Listof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28lib._racket%2Fprivate%2Fmap..rkt%29._map%29%29">map</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-circular-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"D"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"A"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktVal">"A"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"B"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"C"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktVal">"D"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-&gt;</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Void</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-split</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-read</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">append*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">all-circular-shifts*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-lines</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-display</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">alphabetize</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">all-shifts</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">test</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._open-output-string%29%29">open-output-string</a></span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">make-temporary-file</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._with-output-to-file%29%29">with-output-to-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Filesystem.html#%28def._%28%28quote._~23~25kernel%29._delete-file%29%29">delete-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">tmpfile</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">check-equal?</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stringport.html#%28def._%28%28quote._~23~25kernel%29._get-output-string%29%29">get-output-string</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">string-join</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktPn">(</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"if</span><span class="hspace">&nbsp;</span><span class="RktVal">this</span><span class="hspace">&nbsp;</span><span class="RktVal">imagine"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if</span><span class="hspace">&nbsp;</span><span class="RktVal">this"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"this</span><span class="hspace">&nbsp;</span><span class="RktVal">imagine</span><span class="hspace">&nbsp;</span><span class="RktVal">if"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"write</span><span class="hspace">&nbsp;</span><span class="RktVal">took</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktVal">weeks</span><span class="hspace">&nbsp;</span><span class="RktVal">to\n"</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"\n"</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/module.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._module%2B%29%29">module+</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">main</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket/cmdline</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">Parameterof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Any</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28def._%28%28quote._~23~25kernel%29._make-parameter%29%29">make-parameter</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">command-line</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:program</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"kwic</span><span class="hspace">&nbsp;</span><span class="RktVal">index"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:once-each</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">[</span><span class="RktPn">(</span><span class="RktVal">"-o"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"--output"</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">user-supplied</span><span class="hspace">&nbsp;</span><span class="RktCmt">input</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"Write</span><span class="hspace">&nbsp;</span><span class="RktVal">output</span><span class="hspace">&nbsp;</span><span class="RktVal">to</span><span class="hspace">&nbsp;</span><span class="RktVal">file"</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">*output-to*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">update</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameter</span><span class="hspace">&nbsp;</span><span class="RktCmt">*output-to*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:args</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">file-name</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">*output-to*</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">;</span><span class="hspace">&nbsp;</span><span class="RktCmt">read</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">parameter</span><span class="hspace">&nbsp;</span><span class="RktCmt">*output-to*</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/if.html#%28form._%28%28quote._~23~25kernel%29._if%29%29">if</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3f%29%29">string?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/file-ports.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._open-output-file%29%29">open-output-file</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">#:exists</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quote.html#%28form._%28%28quote._~23~25kernel%29._quote%29%29">'</a></span><span class="RktSym">replace</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29">parameterize</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._current-output-port%29%29">current-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">kwic-index</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym">cast</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">file-name</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">Path-String</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/when_unless.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._when%29%29">when</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/strings.html#%28def._%28%28quote._~23~25kernel%29._string~3f%29%29">string?</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">output-to</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/port-ops.html#%28def._%28%28quote._~23~25kernel%29._close-output-port%29%29">close-output-port</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">out-port</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">Sample interactions: +</div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym">kwic</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index:</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">expects</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">1</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">&lt;file-name&gt;</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">on</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">the</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">command</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">line</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/quasiquote.html#%28form._%28%28quote._~23~25kernel%29._unquote%29%29">,</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">given</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">arguments</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta">&#160;</span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">echo</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">"It</span><span class="hspace">&nbsp;</span><span class="RktVal">is</span><span class="hspace">&nbsp;</span><span class="RktVal">a</span><span class="hspace">&nbsp;</span><span class="RktVal">truth</span><span class="hspace">&nbsp;</span><span class="RktVal">universally</span><span class="hspace">&nbsp;</span><span class="RktVal">acknowledged"</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">pride-and-prejudice.txt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">racket</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">kwic.rkt</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-o</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">pride-and-prejudice.txt</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3e%29%29">&gt;</a></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">wc</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym"><span class="nobreak">-l</span></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktVal">6</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">index.out</span><span class="RktMeta"></span></td></tr></tbody></table></div></div> + +<h1><a name="(part._.Closing)"></a>Closing</h1> + +<p>We started with functions, wrote (and quarantined) unit tests, + reinforced our design with types, and added a command-line interface. +Going forward we could add <a href="http://docs.racket-lang.org/scribble/index.html">Scribble</a> documentation and share our work as a <a href="http://pkgn.racket-lang.org/">package</a>.</p> + +<p></p> + +<div class="SIntrapara">For more on building languages with Racket: +</div> + +<div class="SIntrapara"> + <ul> + <li> + <p><a href="http://www.hashcollision.org/brainfudge/">Fudging up a Racket (html)</a></p></li> + <li> + <p><a href="http://dl.acm.org/authorize?6529547">Creating Languages in Racket (pdf)</a></p></li> + <li> + <p><a href="http://www.ccs.neu.edu/home/matthias/manifesto/">The Racket Manifesto (html)</a></p></li> + <li> + <p><a href="http://www.terohasu.net/hasu-flatt--els16--preprint.pdf">Source-to-Source Compilation via Submodules (pdf)</a></p></li></ul></div> + + Tutorial: Racket FFI, part 3 + http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-07-11-tutorial-racket-ffi-part-3 + Mon, 11 Jul 2016 17:33:40 UT + Asumu Takikawa + +<p>This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a> +and part 2 +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a>.</p> + +<p>In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.</p> +<!--more--> + +<p>As in the second post, let&rsquo;s start with some prologue code that establishes +the definitions from the previous two posts. But first, I&rsquo;m getting tired of +writing the <span class="stt">#:c-id identifier</span> notation for the underscored C function +names.</p> + +<p>Instead, let&rsquo;s use a third-party package that I wrote that lets you avoid the +boilerplate. To install the package, you can either invoke the following +incantation in a command-line:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">$</span><span class="hspace">&nbsp;</span><span class="RktMeta">raco</span><span class="hspace">&nbsp;</span><span class="RktMeta">pkg</span><span class="hspace">&nbsp;</span><span class="RktMeta">install</span><span class="hspace">&nbsp;</span><span class="RktMeta">ffi-definer-convention</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>or you can just execute the following snippet in Racket:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pkg</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">pkg-install-command</span><span class="hspace">&nbsp;</span><span class="RktPn">#:skip-installed</span><span class="hspace">&nbsp;</span><span class="RktVal">#t</span><span class="hspace">&nbsp;</span><span class="RktVal">"ffi-definer-convention"</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://download.racket-lang.org/releases/7.9/catalog/</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Resolving "ffi-definer-convention" via https://pkgs.racket-lang.org</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">Downloading repository git://github.com/takikawa/racket-ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: version: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: platform: x86_64-linux [3m]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: target machine: racket</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: installation name: 7.9</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: variants: 3m</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main collects: /usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: collects paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/collects</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main pkgs: /usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: pkgs paths: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/pkgs</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: links files: </span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/usr/share/racket/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">/root/.local/share/racket/7.9/links.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: main docs: /usr/share/racket/doc</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- updating info-domain tables ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: updating: /root/.local/share/racket/7.9/share/info-cache.rktd</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- pre-installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing foreign libraries --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing shared files ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- compiling collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- parallel build using 4 jobs ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:56]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 making: &lt;pkgs&gt;/ffi-definer-convention</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- creating launchers --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing man pages --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- building documentation --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:20:57]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 running: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 3 rendering: &lt;pkgs&gt;/ffi-definer-convention/ffi-definer-convention.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 2 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/local-redirect.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 1 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/release.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/search.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: 0 rendering: &lt;pkgs&gt;/racket-index/scribblings/main/user/start.scrbl</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- installing collections --- </span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">raco setup: --- post-installing collections ---</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktOut">[22:21:03]</span></p></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This will install the package and compile its contents. If you&rsquo;re curious, the +docs for the package are available +<a href="http://docs.racket-lang.org/ffi-definer-convention/index.html">here</a>.</p> + +<p><span style="font-weight: bold">Note:</span> if you&rsquo;ve never installed a package before, you may want to glance +at the +<a href="http://docs.racket-lang.org/pkg/getting-started.html">package system docs</a>. +A tl;dr of packages is that they bundle Racket <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/collects.html#%28tech._collection%29"><span class="techinside">collections</span></a>, which are +sets of modules that you can refer to in a location-independent fashion such as +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> or <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28mod-path._racket%2Flist%29"><span class="RktSym">racket/list</span></a></span>.</p> + +<p>Anyhow, here&rsquo;s the prologue code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">avoid conflict with below</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._except-in%29%29">except-in</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">define-ffi-definer</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the new 3rd-party pkg</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi-definer-convention</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">describes how to transform from</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Racket to C ids</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:make-c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">convention:hyphen-&gt;underscore</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">note lack of #:c-id keyword arguments</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">(_cairo_t -&gt; Void) -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">do some drawing and give us the pict</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">f</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">f</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Notice that the <span class="RktWrap"><span class="RktSym">define-cairo</span></span> forms don&rsquo;t have any <span class="stt">#:c-id</span> keywords +anymore. Instead, the prologue code uses an overriden <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> +from my package that supports a <span class="stt">#:make-c-id</span> keyword that lets you specify +a naming convention to follow.</p> + +<p>Also, instead of creating a single bitmap and drawing into it, we now have a +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> function that takes a drawing function. When called, +<span class="RktWrap"><span class="RktSym">do-cairo</span></span> will call the given function with a new bitmap object and +return the result.</p> + +<p>Now let&rsquo;s get to the main point of this blog post. Let&rsquo;s say that we want to play +with Cairo <a href="https://www.cairographics.org/manual/cairo-Paths.html">path</a> +objects this time. A path is +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-t">defined</a> +as a struct with the following structure:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_status_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">status</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_data</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>To manipulate paths, we want to define a FFI C type that corresponds to this +struct definition. But before that, it&rsquo;s +useful to define C types for the types of values in the path struct&rsquo;s fields. First, +let&rsquo;s specify that a <span class="stt">cairo_status_t</span> is an integer type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>It&rsquo;s actually an enum, but for the examples in this post we don&rsquo;t care about +distinguishing different statuses. Next, the data field of a path struct is an +array of +<a href="https://www.cairographics.org/manual/cairo-Paths.html#cairo-path-data-t">path data objects</a>. +Each path data object is a <span class="stt">cairo_path_data_t</span>, +which is specified with a C union:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">union</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">_cairo_path_data_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_path_data_type_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">type</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">length</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">header</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">point</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Helpfully, the FFI library comes with support for unions with the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span></span> type constructor. The constructor takes arbitrarily +many arguments, one for each sub-case in the union. It&rsquo;s pretty +straightforward to specify this type too:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the path data type is just an enum</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">move-to</span><span class="hspace">&nbsp;</span><span class="RktVal">line-to</span><span class="hspace">&nbsp;</span><span class="RktVal">curve-to</span><span class="hspace">&nbsp;</span><span class="RktVal">close-path</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__union%29%29">_union</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the header case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_type_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the point case</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>There&rsquo;s a new type constructor here so let me explain that first. +The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span></span> constructor translates between a C struct +and a fixed-length list of C objects on the Racket side. Unlike +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span>, this constructor doesn&rsquo;t define any selectors +or anything like that. Instead, you can manipulate the struct as an +ordinary list.</p> + +<p>Each of the path data structs in the path data array will be manipulated with +the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> type. Union types are a bit cumbersome unfortunately +because the programmer has to distinguish the cases in the union manually +on the Racket-side. Let me illustrate this with some code:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">create a union from a list of doubles</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">1.3</span><span class="hspace">&nbsp;</span><span class="RktVal">5.8</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">source type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">target type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-union-val</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;union&gt;</span></p></td></tr></tbody></table></div> + +<p>This snippet first construct a union object (via the <span class="RktWrap"><span class="RktSym">_cairo_path_data_t</span></span> +type) using a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Miscellaneous_Support.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cast%29%29">cast</a></span></span>. A cast is an operation that lets you coerce +from one C type to another. We use it in this example since it&rsquo;s an easy way +to generate a union object.</p> + +<p>The second line shows that a union prints as an opaque object. You can&rsquo;t do +anything with a union in Racket unless you project it to one of the sub-cases with +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></span> function. +This projection is <span class="emph">unsafe</span>, in the sense that if you don&rsquo;t know which of +the sub-cases in the union is the correct one, you will get potentially non-sensical +data out of the union.</p> + +<p>More concretely, let&rsquo;s see what happens if we try to extract a value out of the +union both correctly and incorrectly:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">correct (matches construction)</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">cases are zero-indexed and ordered as written</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(1.3 5.8)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">incorrect, error</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-union-val</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">enum:int-&gt;_cairo_path_data_type_t: expected a known</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">#&lt;ctype:ufixint&gt;, got: 3435973837</span></p></td></tr></tbody></table></div> + +<p>Note that in the incorrect case we get an error saying that the FFI failed +to convert the C value to a Racket value following the given C type. We were +lucky in this case, but in general you can have silent failures where the +data is nonsense.</p> + +<p>With union types like these, there is usually some way to figure out which case +of the union you are in. This may be accomplished in C using an extra struct +field or a variable that indicates the variant. Alternatively, there may be some +set order that cases appear in data structures.</p> + +<p>With this Cairo API in particular, the position of the elements in the array +tells you which of the union cases it&rsquo;s in. The array always starts with a +header element, and then follows with some number of data elements (the exact +number is determined by the type indicated in the header). We can therefore +reference the appropriate element of the union based on this ordering.</p> + +<p>So before moving on, let&rsquo;s recap: so far we have made C types that describe +the data elements in a cairo path with unions. Next we&rsquo;ll figure out how to +deal with the array itself.</p> + +<h1><a name="(part._.Some_low-level_operations)"></a>Some low-level operations</h1> + +<p>Since we still don&rsquo;t have a C type for <span class="stt">cairo_path_t</span>, let&rsquo;s go ahead +and make a simple one where we punt on the work of specifying the array +type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__list-struct%29%29">_list-struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_status_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In this type, we have specified the array as a bare <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span>. +For some added safety, we could also use something like +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__cpointer%29%29">_cpointer</a></span><span class="stt"> </span><span class="RktVal">'</span><span class="RktVal">cairo_status_t</span><span class="RktPn">)</span></span>, which sets up a tagged pointer +type like we saw in the first blog post with +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>.</p> + +<p>We&rsquo;ve seen the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type before, but haven&rsquo;t actually done +anything with values of those types except pass them around as arguments. +It turns out it is possible to do a bit more with pointers.</p> + +<p>Before we get to that, let&rsquo;s go ahead and set up an FFI binding for +<span class="stt">cairo_copy_path</span> so that we can obtain a path struct to manipulate:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Do stuff to make the current</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">path non-empty</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Get the current path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/set_.html#%28form._%28%28quote._~23~25kernel%29._set%21%29%29">set!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Stroke clears the path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">so do it last</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">a-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<p>Note that <span class="RktWrap"><span class="RktSym">cairo-copy-path</span></span> gives us a pointer to a path struct +rather than a path struct directly. Because of that, we need to know +how to manipulate pointers. +The most useful function for pointers is <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span>, which +lets you dereference a pointer and access it at some concrete C type.</p> + +<p><span style="font-weight: bold">Note:</span> the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> function also takes an optional +offset argument which we will be used in an example later.</p> + +<p>For example, we can use <span class="RktWrap"><span class="RktSym">a-path</span></span> as a <span class="RktWrap"><span class="RktSym">_simple_cairo_path_t</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">a-path</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">simple-path</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(0 #&lt;cpointer&gt; 8)</span></p></td></tr></tbody></table></div> + +<p>And now we have a Racket representation of the struct that the pointer +points to. Now notice that the data array field of the struct is also +a pointer as we specified earlier. To convert this to a more useful form, +we can use <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> again with an array type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">array</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">length field</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">simple-path</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">array</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(#&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt; #&lt;union&gt;)</span></p></td></tr></tbody></table></div> + +<p>The elements of the array are all unions, as we would expect. This is +a bit annoying to use though. We have to know the structure of the +array and reference the correct variant appropriately:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(move-to 2)</span></p></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(50.0 50.0)</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">nonsense data here, wrong union case</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">third</span><span class="hspace">&nbsp;</span><span class="RktSym">array</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">'(4.2439915824246e-314 0.0)</span></p></td></tr></tbody></table></div> + +<p>One thing we could do is write a helper function that converts this array +into a more useful format. It would look at each header, and then consume +the number of data elements specified in the header element (e.g., 1 +in the example above because the length includes the header) +and convert them appropriately.</p> + +<p>An alternative is to define a <span class="emph">custom C type</span> that handles all of +this conversion automatically for us, so that as a user of the Cairo +FFI bindings we don&rsquo;t need to think about applying helper functions and +dereferencing pointers.</p> + +<h1><a name="(part._.Custom_.C_types)"></a>Custom C types</h1> + +<p>I briefly remarked on how to create custom C types in the first blog post, +but let me go over that again in more detail. A custom C type is constructed +by providing a base C type to use along with two conversion functions. +The first function converts from a Racket value to a value that fits the +base C type. The second converts in the other direction from a value of +the base C type to a Racket value.</p> + +<p>In this way, it&rsquo;s possible to conduct interesting conversions, such as +dereferencing union objects automatically.</p> + +<p>Now let&rsquo;s make a custom C type for Cairo paths that will represent the +data elements as a sequence in which each item in the sequence is a list +with an action symbol followed by the data elements for that action.</p> + +<p>First, we&rsquo;ll start by defining a struct type for the Racket representation +of Cairo paths:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define-struct.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._struct%29%29">struct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ptr</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:property</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._prop~3asequence%29%29">prop:sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">p</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The representation will store one field <span class="RktWrap"><span class="RktSym">ptr</span></span> which, as the name +suggests, will store a pointer value. We&rsquo;ll see what to do with this +pointer later.</p> + +<p>This definition uses a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/structprops.html#%28tech._structure._type._property%29"><span class="techinside">structure type property</span></a> to make instances of +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> automatically work as sequences. This means that you +can iterate over them with a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span></span> loop or apply <span class="RktWrap"><span class="RktSym">sequence-ref</span></span> +on them. The property takes a function that takes an instance of the struct +type itself (here <span class="RktWrap"><span class="RktSym">p</span></span>) and that returns a sequence.</p> + +<p>We&rsquo;ll later define the <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span> function that will actually +construct the relevant sequence for us. For now, let&rsquo;s see how to construct +the C type given this struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/let.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%29%29">let</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract pointer out of representation</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">racket-&gt;c</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">rkt</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Just apply the Racket constructor</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">c-&gt;racket</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">cobj</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">racket-&gt;c</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">c-&gt;racket</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The base type for this <span class="RktWrap"><span class="RktSym">_cairo_path_t</span></span> is a <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> type. Since +the Cairo API returns pointers to new path values, it&rsquo;s hard to avoid using some +kind of pointer type as the base type here.</p> + +<p>This definition right-hand-side defines the two conversion functions between +Racket and C. Both are very simple because of how we&rsquo;ve set up the representation. +In the Racket to C case, we simply extract the pointer field of the struct. In +the other direction, we just stuff the pointer into a struct.</p> + +<p>The real work is done by the helper function that makes a +<span class="RktWrap"><span class="RktSym">cairo-path</span></span> instance work as a sequence.</p> + +<p>Starting top-down, let&rsquo;s look at the definition of <span class="RktWrap"><span class="RktSym">in-cairo-path</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Cairo-Path -&gt; Sequence</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">in-cairo-path</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-path-ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">match-define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29.__%29%29">_</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pp</span><span class="hspace">&nbsp;</span><span class="RktSym">_simple_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/values.html#%28def._%28%28quote._~23~25kernel%29._values%29%29">values</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._array-ptr%29%29">array-ptr</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">#f</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The first thing the function does is extract the pointer out of the +representation, and then immediately calls <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span></span> on it. This +lets us manipulate the C path struct using the simple representation we +defined in the first part of the blog post.</p> + +<p><span style="font-weight: bold">Note:</span> in case you&rsquo;re not very familiar with Racket pattern matching, the +<span class="RktWrap"><span class="RktSym">match-define</span></span> form lets you define potentially multiple variables +using a pattern, similar to Haskell or OCaml&rsquo;s <span class="stt">let</span> statement. +The first argument clause +is a pattern and the second is an expression to match on. Check it out +in the +<a href="http://docs.racket-lang.org/reference/match.html#%28form._%28%28lib._racket%2Fmatch..rkt%29._match-define%29%29">docs</a>.</p> + +<p>After extracting the array pointer and the array length from the +path value, we pass them onto some helper functions that define the +sequence. The usual way to define a new kind of sequence is to use the +<span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> function. Essentially, <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> +takes a bunch of arguments that specify how to get the an element of +a sequence, how to advance a sequence, how to start, and how to end +the sequence.</p> + +<p><span style="font-weight: bold">Note:</span> technically <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._make-do-sequence%29%29">make-do-sequence</a></span></span> actually takes a thunk which +produces a number of values. These values are effectively like arguments +though. The reason why it&rsquo;s a thunk is that you may wish to +run some initialization code that runs when the sequence is started +(e.g., imagine opening a network connection), and your sequence functions +(like advancing the sequence) may depend on the result of that +initialization code.</p> + +<p>In our case, we supply some curried functions that can extract elements +out of the underlying C array. Here is the <span class="RktWrap"><span class="RktSym">pos-&gt;element</span></span> function +and its helpers:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer -&gt; Integer -&gt; Element</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">pos-&gt;element</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Extract the data path header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">first</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Length includes header, so subtract 1</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._sub1%29%29">sub1</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._add1%29%29">add1</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos*</span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._cons%29%29">cons</a></span><span class="hspace">&nbsp;</span><span class="RktSym">type</span><span class="hspace">&nbsp;</span><span class="RktSym">points</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">CPointer Integer Integer -&gt; (Listof Data)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">get-points</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29">for/list</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/sequences.html#%28def._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._in-range%29%29">in-range</a></span><span class="hspace">&nbsp;</span><span class="RktSym">num-points</span><span class="RktPn">)</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_path_data_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">offset argument</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">1</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This code encodes the API usage protocol that Cairo specifies, where each +header element in the path is followed by some number of data elements. +Each header specifies the length, so we can loop in <span class="RktWrap"><span class="RktSym">get-points</span></span> +from the position after the header until we reach the given length. At +each point, we dereference the appropriate union element.</p> + +<p>Advancing the sequence is simpler, since all we need to do is some arithmetic +on the length given by header elements:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">(</span><span class="RktSym">next-pos</span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">header</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Union_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._union-ref%29%29">union-ref</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._ptr-ref%29%29">ptr-ref</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ptr</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_data_t</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">second</span><span class="hspace">&nbsp;</span><span class="RktSym">header</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">len</span><span class="hspace">&nbsp;</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Note that determining the end of the sequence is very easy. It&rsquo;s just +a matter of comparing the current position to the total length given in the +path struct, encoded in the expression <span class="RktWrap"><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym">pos</span><span class="RktPn">)</span><span class="stt"> </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3c%29%29">&lt;</a></span><span class="stt"> </span><span class="RktSym">pos</span><span class="stt"> </span><span class="RktSym">len</span><span class="RktPn">)</span><span class="RktPn">)</span></span>.</p> + +<p>Now we can try using a path as a sequence:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-copy-path</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_path_t</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">do-cairo</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/lambda.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._~ce~bb%29%29">&#955;</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">206.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="hspace">&nbsp;</span><span class="RktVal">115.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-copy-path</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Using path as a sequence</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/for.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%29%29">for</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">elem</span><span class="hspace">&nbsp;</span><span class="RktSym">path</span><span class="RktPn">]</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/Writing.html#%28def._%28%28lib._racket%2Fprivate%2Fmisc..rkt%29._displayln%29%29">displayln</a></span><span class="hspace">&nbsp;</span><span class="RktSym">elem</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0"> + <tbody> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 50.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (206.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(move-to (50.0 206.0))</span></p></td></tr> + <tr> + <td> + <p><span class="RktOut">(line-to (115.0 115.0))</span></p></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>Notice how the sequence prints out as an intuitive list of commands +instead of a bunch of opaque union values as we saw before when using +the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Array_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__array%2Flist%29%29">_array/list</a></span></span> type.</p> + +<p>That concludes part 3 of the FFI tutorial. Hopefully you&rsquo;re now equipped +to deal with union types and custom C types. If not, see the +<a href="http://docs.racket-lang.org/foreign/index.html">FFI reference</a> +for more details on +<a href="http://docs.racket-lang.org/foreign/C_Union_Types.html">unions</a> +and +<a href="http://docs.racket-lang.org/foreign/ctype.html">custom C types</a>.</p> + +<p><span class="emph">Thanks to Ben Greenman for suggestions/feedback and to Sam +Tobin-Hochstadt for suggesting to cover union types!</span></p> + + Tutorial: Racket FFI, Part 2 + http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-29-tutorial-racket-ffi-part-2 + Wed, 29 Jun 2016 18:48:17 UT + Asumu Takikawa + +<p>This is part 2 of my tutorial on using the Racket FFI. If you haven&rsquo;t read +part 1 yet, you can find it +<a href="http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/">here</a>. +<span style="font-weight: bold">Update:</span> part 3 is also now available +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.</p> +<!--more--> + +<p>First, here&rsquo;s the core code from part 1 condensed and re-arranged into an +example that you can copy and paste into your definitions area:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/guide/Module_Syntax.html#%28part._hash-lang%29"><span class="RktMod">#lang</span></a><span class="hspace">&nbsp;</span><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/index.html"><span class="RktSym">racket</span></a></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">ffi/unsafe/define</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">bitmap magic</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">C types</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the foreign functions</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_create</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">Bitmap -&gt; Pict</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a helper for displaying the bitmap</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<h1><a name="(part._.Dashes_and_array_arguments)"></a>Dashes and array arguments</h1> + +<p>To start off, let&rsquo;s look at another C example from the Cairo +<a href="https://www.cairographics.org/samples/">samples page</a>. +This time we will look at the "dash" example, which has a +use of an input array:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktPn">]</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">ink</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">skip*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">sizeof</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">)</span><span class="RktMeta">/sizeof</span><span class="RktPn">(</span><span class="RktMeta">dashes</span><span class="RktPn">[</span><span class="RktVal">0</span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">=</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">50.0</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">dashes,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">ndash,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">10.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">25.6</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_rel_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">-</span><span class="RktVal">102.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_curve_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">230.4</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">51.2</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">128.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The most interesting function here is <span class="stt">cairo_set_dash</span>, which takes an +array argument. The only other new functions are <span class="stt">cairo_rel_line_to</span> +and <span class="stt">cairo_curve_to</span> which have very straightforward C types:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-rel-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_rel_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-curve-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_curve_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Meanwhile, the C type signature for <span class="stt">cairo_set_dash</span> from the Cairo +docs looks like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_dash</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">int</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">num_dashes,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">offset</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Something to note about the arguments is that <span class="stt">num_dashes</span> +encodes the length of the array <span class="stt">dashes</span>. This will come up later when +we want to make the C type for this function more convenient.</p> + +<p>On the Racket side, it&rsquo;s natural to represent the array of dashes as either a +list or <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/vectors.html#%28tech._vector%29"><span class="techinside">vector</span></a> of numbers. Given that, a fairly literal translation of +the C type above might look like the following:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This type includes a type constructor we haven&rsquo;t seen yet: <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span>. +This is a so-called <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28tech._custom._function._type%29"><span class="techinside">custom function type</span></a> that has special meaning +inside of a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type. It lets you convert between a Racket list and +a C array. Since arrays are often used for both input and output of a C function, +the constructor requires you to specify the mode in which you are using the +array.</p> + +<p>Since we only want to provide a list from the Racket side to C, we&rsquo;ll use +the <span class="RktWrap"><span class="RktSym">i</span></span> input mode. We can then call the function like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">4</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Note that because of how we defined the type of <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> we had to +provide the length of the input list as a separate argument! This seems pretty +silly since it&rsquo;s very easy to get the length of a Racket list and because this +is a likely source of mistakes. It would be preferable to compute the length +argument automatically.</p> + +<p>Luckily, the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type constructor actually lets you do this with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">name</span><span class="stt"> </span><span class="RktSym">:</span><span class="stt"> </span><span class="RktSym">type</span><span class="RktPn">)</span></span> syntax for naming arguments in combination with the +<span class="RktWrap"><span class="RktPn">(</span><span class="RktSym">type</span><span class="stt"> </span><span class="RktVar">=</span><span class="stt"> </span><span class="RktSym">expr</span><span class="RktPn">)</span></span> syntax for supplying computed arguments:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-dash</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">name this argument for later uses in the type</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span><span class="hspace">&nbsp;</span><span class="RktSym">i</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a computed argument position</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._~3d%29%29">=</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._length%29%29">length</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_dash</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>When a computed argument is specified with a <span class="RktWrap"><span class="RktVar">=</span></span>, it&rsquo;s not necessary to provide +the argument on the Racket side. So <span class="RktWrap"><span class="RktSym">cairo-set-dash</span></span> is now an arity 3 +function that can be called like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/pairs.html#%28def._%28%28quote._~23~25kernel%29._list%29%29">list</a></span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This means we&rsquo;ll never make a mistake in passing the length argument to Cairo. +Just as an aside, it&rsquo;s also possible to use Racket vectors instead of lists by using +the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__vector%29%29">_vector</a></span></span> type constructor.</p> + +<p>Putting it all together, we can reproduce the dashes example like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktVal">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-5</span>0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-dash</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">dashes</span><span class="hspace">&nbsp;</span><span class="RktSym">offset</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">10.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">25.6</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-rel-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal"><span class="nobreak">-1</span>02.4</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-curve-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span><span class="hspace">&nbsp;</span><span class="RktVal">230.4</span><span class="hspace">&nbsp;</span><span class="RktVal">51.2</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td></td></tr></tbody></table></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Result_arguments_and_.C_structs)"></a>Result arguments and C structs</h1> + +<p>For some more advanced FFI hacking, let&rsquo;s consider the problem of drawing some +text into a predetermined space. In particular, we have our usual 256x256 +bitmap that we want to draw some text into:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Our challenge is to make a Racket function that takes a string (let&rsquo;s +assume we can draw it in one line) and draws it into this bitmap. +Since we are taking an arbitrary string, we will need to figure out +how to scale the text to fit. To make it simple, let&rsquo;s just scale the +text to fit the width and assume the height will be okay.</p> + +<p>To implement the key step of measuring the text size, we can use the +<a href="https://www.cairographics.org/manual/cairo-text.html#cairo-text-extents"><span class="stt">cairo_text_extents</span></a> +function. Its type signature is as follows:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">const</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">char</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*utf8,</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*extents</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>The interesting part of this signature is that +<a href="https://www.cairographics.org/manual/cairo-cairo-scaled-font-t.html#cairo-text-extents-t"><span class="stt">cairo_text_extents_t</span></a> +is a struct type:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">from</span><span class="hspace">&nbsp;</span><span class="RktCmt">the</span><span class="hspace">&nbsp;</span><span class="RktCmt">Cairo</span><span class="hspace">&nbsp;</span><span class="RktCmt">docs</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">typedef</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">struct</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">{</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_bearing</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">height</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y_advance</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktPn">}</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_text_extents_t</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>We haven&rsquo;t yet seen how to handle C structs with the FFI, but it&rsquo;s not +too tricky. Support for C structs comes built-in and will look familiar +if you&rsquo;re used to Racket structs. We can directly translate the documented +definition above into a <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span></span> declaration:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the leading underscore is mandatory</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/C_Struct_Types.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cstruct%29%29">define-cstruct</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktPn">[</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">width</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">height</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">x-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">y-advance</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="RktPn">]</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This declaration does a couple of things. First, it defines a bunch of handy C types +related to the struct for us:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">pointer to struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">allows NULL pointer</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">_cairo_text_extents_t-pointer/null</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;ctype&gt;</span></p></td></tr></tbody></table></div> + +<p>Along with functions that look like regular Racket struct operations:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a struct constructor</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:make-cairo_text_extents_t&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field selector</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:cairo_text_extents_t-width&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a predicate for the struct</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">cairo_text_extents_t?</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:^TYPE?&gt;</span></p></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a field mutation function</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">set-cairo_text_extents_t-width!</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;procedure:set-cairo_text_extents_t-width!&gt;</span></p></td></tr></tbody></table></div> + +<p>With the struct type defined, it&rsquo;s easy to come up with a rudimentary +interface for <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">_cairo_text_extents_t-pointer</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>In order to actually use this function, we need to create a text extents struct +and provide it as a pointer. Conveniently, the FFI treats instances of C structs +as pointers so this is pretty straightforward:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-cairo_text_extents_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="hspace">&nbsp;</span><span class="RktVal">0.0</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>This style of programming feels awfully imperative though. Since we&rsquo;re in a +functional language, it would be nice to avoid the manual creation of the struct. +We can define an alternate version of the <span class="RktWrap"><span class="RktSym">cairo-text-extents</span></span> FFI wrapper +by combining named arguments, a new <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> type constructor, +and a neat feature of <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> that lets you customize +the return result:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-text-extents*</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">named args and _ptr</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">[</span><span class="RktSym">ext</span><span class="hspace">&nbsp;</span><span class="RktSym">:</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span><span class="hspace">&nbsp;</span><span class="RktSym">o</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_text_extents_t</span><span class="RktPn">)</span><span class="RktPn">]</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">the return result of the C function</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">custom return result for the wrapper</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">ext</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_text_extents</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span> constructor works like the <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__list%29%29">_list</a></span></span> constructor we saw +earlier in this blog post but typically for a single object. Since we are passing +in a value to use as an output, we specify the <span class="RktWrap"><span class="RktSym">o</span></span> mode to <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__ptr%29%29">_ptr</a></span></span>. +In output mode, this type will automatically allocate a new instance of the type +(using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_pointer-funcs.html#%28def._%28%28quote._~23~25foreign%29._malloc%29%29">malloc</a></span></span> function) and arrange for it to be passed in as +a pointer.</p> + +<p>The strangest part of this example is that there are now two uses of the +<span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span></span> form! By providing a second arrow, we can customize what the FFI wrapper +returns. The expression to the right of the second arrow is just Racket code that can +reference previously named arguments. The result of evaluating this +expression is used instead of the normal return result for calls to the +wrapped function. In this case, we just return the struct that was allocated for us.</p> + +<p>Using this new version of the wrapper is much simpler:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">"hello world"</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <p><span class="RktRes">54.0</span></p></td></tr></tbody></table></div> + +<p>With that in hand, it&rsquo;s pretty easy to write the function we set out to write:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-show-text</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/String_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__string%29%29">_string</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_show_text</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-scale</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">String -&gt; Void</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">draws a string scaled horizontally</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">20</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="hspace">&nbsp;</span><span class="RktVal">2.0</span><span class="RktPn">)</span><span class="hspace">&nbsp;</span><span class="RktVal">128.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">extents</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-text-extents*</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-x-bearing</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo_text_extents_t-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktSym">extents</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2F%29%29">/</a></span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._-%29%29"><span class="nobreak">-</span></a></span><span class="hspace">&nbsp;</span><span class="RktVal">256.0</span><span class="hspace">&nbsp;</span><span class="RktSym">padding</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/generic-numbers.html#%28def._%28%28quote._~23~25kernel%29._%2B%29%29">+</a></span><span class="hspace">&nbsp;</span><span class="RktSym">x-bearing</span><span class="hspace">&nbsp;</span><span class="RktSym">width</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-scale</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="hspace">&nbsp;</span><span class="RktSym">scale</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-show-text</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-ctx</span><span class="hspace">&nbsp;</span><span class="RktSym">str</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>And to conclude part 2 of this tutorial, here&rsquo;s an example use +of the new <span class="RktWrap"><span class="RktSym">fit-text</span></span> function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">fit-text</span><span class="hspace">&nbsp;</span><span class="RktVal">"Saluton, Mondo / Hallo, mundo"</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">show</span><span class="hspace">&nbsp;</span><span class="RktSym">txt-bt</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + + Tutorial: Using Racket's FFI + http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/?utm_source=tutorial&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2016-06-27-tutorial-using-racket-s-ffi + Mon, 27 Jun 2016 16:22:11 UT + Asumu Takikawa + +<p><span style="font-weight: bold">Update:</span> this post is now part of a series. Part 2 is +<a href="http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/">here</a> +and part 3 is +<a href="http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/">here</a>.</p> + +<p>I&rsquo;ve seen several people ask for a tutorial on Racket&rsquo;s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +<a href="http://docs.racket-lang.org/foreign/index.html">documentation</a> +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.</p> + +<p>With that in mind, this blog post will provide a step-by-step tutorial +for Racket&rsquo;s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.</p> +<!--more--> + +<p>Before getting into the details, I wanted to note that the FFI library +is based on the work of Eli Barzilay and Dmitry Orlovsky. They have +a Scheme Workshop <a href="http://www.ccs.neu.edu/racket/pubs/scheme04-bo.pdf">paper</a> +that you can read if you&rsquo;re curious about the design.</p> + +<p>The tutorial will focus on using the <a href="https://www.cairographics.org/">Cairo</a> +graphics library, mainly because it comes bundled with Racket.</p> + +<p>To start, let&rsquo;s aim to reproduce the output of the "multi segment caps" +C sample code on Cairo&rsquo;s +<a href="https://www.cairographics.org/samples/">samples page</a>:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">75.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">125.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">50.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">200.0</span><span class="RktMeta">,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">175.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">&#160;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktVal">30.0</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">CAIRO_LINE_CAP_ROUND</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>In order to actually draw this example to the screen, we will need a Cairo +surface to draw on. So here is some boilerplate for you to execute before we +actually play with the FFI:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">racket/draw</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">make-bitmap</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="hspace">&nbsp;</span><span class="RktVal">256</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">send</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="hspace">&nbsp;</span><span class="RktSym">get-handle</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>This uses the Racket drawing library <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/draw/index.html"><span class="RktSym">racket/draw</span></a></span> to construct +a bitmap object that we&rsquo;ll draw on. The <span class="RktWrap"><span class="RktSym">get-handle</span></span> method just extracts a +low-level Cairo surface value that we can use.</p> + +<p>NB: these code snippets don&rsquo;t come with a <span class="stt">#lang</span> declaration because +they simulate interactions at the REPL/interaction area in DrRacket. +When following along, just copy &amp; paste the snippets into your REPL.</p> + +<p>Our first real step is to import the FFI itself:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>As the module name suggests, the FFI is <span class="emph">unsafe</span> and can cause your Racket process +to segfault. If you&rsquo;re following along in DrRacket, you will want to save your file +frequently.</p> + +<p>Next, we can load the Cairo library to obtain a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28tech._foreign._library._value%29"><span class="techinside">foreign-library value</span></a>, which +is a handle that we use to access C values and functions:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Since Cairo has already been loaded by the Racket process because of the +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/gui/index.html"><span class="RktSym">racket/gui</span></a></span> import earlier, we can supply <span class="RktWrap"><span class="RktVal">#f</span></span> here as an +argument to <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span>. Normally you supply the name of a shared +library file such as <span class="RktWrap"><span class="RktVal">"libcairo"</span></span>:</p> + +<div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"libcairo"</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">"2"</span><span class="hspace">&nbsp;</span><span class="RktVal">#f</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></p></div> + +<p>The last list argument specifies the accepted versions (<span class="RktWrap"><span class="RktVal">#f</span></span> allows +a version-less library). For this post, those details aren&rsquo;t important but +see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._ffi-lib%29%29">ffi-lib</a></span></span> if you&rsquo;re curious.</p> + +<h1><a name="(part._.Extracting_functions)"></a>Extracting functions</h1> + +<p>Since the Racket FFI is a dynamic interface, we can pull out C functions +at run-time using the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> function. The <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></span> +function takes three arguments:</p> + +<ul> + <li> + <p>The name of the value as a string (or symbol or bytestring)</p></li> + <li> + <p>a foreign library value, and</p></li> + <li> + <p>a <a class="techoutside Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/types.html#%28tech._c._type%29"><span class="techinside">C type</span></a>, which is a type description that tells +the FFI how to marshall between Racket and C.</p></li></ul> + +<p>C types are a crucial concept for the FFI. They range from relatively +simple types like <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__int%29%29">_int</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> to more complicated +type constructors such as <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> and <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span>. As you probably noticed, +C types are prefixed with an underscore by convention. You can also define +your own types by calling <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/ctype.html#%28def._%28%28quote._~23~25foreign%29._make-ctype%29%29">make-ctype</a></span></span> with two functions that +handle marshalling between C and Racket code.</p> + +<p>To make progress with our Cairo code, we need to create a drawing context from +the surface object <span class="RktWrap"><span class="RktSym">bt-surface</span></span> that we defined a while ago. The +relevant function in the Cairo docs is +<a href="https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-create"><span class="stt">cairo_create</span></a>, +which has the following type signature:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktCmt">/*</span><span class="hspace">&nbsp;</span><span class="RktCmt">NB:</span><span class="hspace">&nbsp;</span><span class="RktCmt">this</span><span class="hspace">&nbsp;</span><span class="RktCmt">is</span><span class="hspace">&nbsp;</span><span class="RktCmt">C</span><span class="hspace">&nbsp;</span><span class="RktCmt">code</span><span class="hspace">&nbsp;</span><span class="RktCmt">*/</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_create</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_surface_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*target</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p></p> + +<div class="SIntrapara">To use this function from Racket, we will need to create a C type that describes +its behavior. As you can see, the function takes a pointer to a <span class="stt">cairo_surface_t</span> and +returns a pointer to a <span class="stt">cairo_t</span>. Let&rsquo;s start with a very simple C type +that matches up with this behavior: </div> + +<div class="SIntrapara"> + <div class="SCodeFlow"> + <p><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span></p></div></div> + +<div class="SIntrapara">This type provides very little safety (in particular, it lets you mix up different +kinds of pointers), but it will work as a first step. +Note that the FFI library uses infix arrow notation for its <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span></span> type.</div> + +<p>The following definition shows how to use this type to obtain a foreign +function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>Then we can use <span class="RktWrap"><span class="RktSym">cairo-create</span></span> as an ordinary racket function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktSym">ctx</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#&lt;cpointer&gt;</span></p></td></tr></tbody></table></div> + +<h1><a name="(part._.Interlude__more_type_safety)"></a>Interlude: more type safety</h1> + +<p>Before we move on to completing the Cairo sample, lets consider the safety of the +C type we used again. Since we only specified <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span></span> types, it is easy +to accidentally misuse the function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">You may not want to actually run this</span></td></tr> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">a cairo_t is not a cairo_surface_t</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>To prevent such bad uses, it is good practice to use <span class="emph">tagged</span> pointer types +using <span class="RktWrap"><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span></span>. Here are two example definitions that +correspond to the <span class="stt">cairo_t</span> and <span class="stt">cairo_surface_t</span> types from earlier:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktCmt">;</span><span class="RktCmt">&nbsp;</span><span class="RktCmt">The leading underscores are mandatory</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._define-cpointer-type%29%29">define-cpointer-type</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>We can then redefine <span class="RktWrap"><span class="RktSym">cairo-create</span></span> with a better type, which will +prevent ill-typed calls:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-create</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span><span class="hspace">&nbsp;</span><span class="RktVal">"cairo_create"</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_surface_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">cairo-create</span><span class="hspace">&nbsp;</span><span class="RktSym">bt-surface</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktErr">cairo_surface_t-&gt;C: argument is not non-null</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr">`cairo_surface_t' pointer</span></p></td></tr> + <tr> + <td> + <p><span class="RktErr"></span><span class="hspace">&nbsp;&nbsp;</span><span class="RktErr">argument: #&lt;cpointer:cairo_t&gt;</span></p></td></tr></tbody></table></div> + +<p>Unfortunately our old definition of <span class="RktWrap"><span class="RktSym">ctx</span></span> doesn&rsquo;t have this tag:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#f</span></p></td></tr></tbody></table></div> + +<p>Which means we will see errors if we try to use it in future interactions with +the more precise C type. To get around this, it&rsquo;s also possible to update +existing pointers with a tag like this:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-push-tag%21%29%29">cpointer-push-tag!</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_tagged-pointers.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._cpointer-has-tag~3f%29%29">cpointer-has-tag?</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">cairo_t</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><span class="RktRes">#t</span></p></td></tr></tbody></table></div> + +<p>Executing the tag push above is necessary to get some of the following snippets +to work (if you are following along step-by-step).</p> + +<h1><a name="(part._.Macros_for_reducing_boilerplate)"></a>Macros for reducing boilerplate</h1> + +<p>Now let&rsquo;s start building the FFI bindings for the functions in the Cairo sample. +First, let&rsquo;s go ahead and look at all of the types for the sample functions +from the C API docs:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_move_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_to</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">x,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">y</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_width</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">double</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">width</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_set_line_cap</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr,</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_line_cap_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">line_cap</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr> + <tr> + <td><span class="RktMeta"></span><span class="RktMeta">void</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">cairo_stroke</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktPn">(</span><span class="RktMeta">cairo_t</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktMeta">*cr</span><span class="RktPn">)</span><span class="RktMeta">;</span><span class="RktMeta"></span></td></tr></tbody></table></div> + +<p>Starting with <span class="stt">cairo_move_to</span>, we can set up a definition like we did +with <span class="stt">cairo_create</span> before:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Loading_Foreign_Libraries.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29._get-ffi-obj%29%29">get-ffi-obj</a></span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktVal">"cairo_move_to"</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktSym">cairo-lib</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Pointer_Types.html#%28def._%28%28quote._~23~25foreign%29.__pointer%29%29">_pointer</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>This starts to look awfully verbose once you start writing more of these +definitions. Luckily, the FFI library comes with some definition forms +in the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span> library that help reduce the +verbosity. Here&rsquo;s an alternative definition of <span class="RktWrap"><span class="RktSym">cairo-move-to</span></span> +using the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form from +<span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Defining_Bindings.html"><span class="RktSym">ffi/unsafe/define</span></a></span>.</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">ffi/unsafe/define</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-ffi-definer</span><span class="hspace">&nbsp;</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-lib</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-move-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_move_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>As you can see, the <span class="RktWrap"><span class="RktSym">define-ffi-definer</span></span> form lets you define a +new macro that lets you avoid writing the library value over and over. +If you stick to using C-style identifiers with underscores (e.g., +<span class="RktWrap"><span class="RktSym">cairo_move_to</span></span>) you also don&rsquo;t need to supply the C name either.</p> + +<p>The definitions for <span class="stt">cairo_line_to</span>, <span class="stt">cairo_set_line_width</span>, and +<span class="stt">cairo_stroke</span> aren&rsquo;t very interesting, so I&rsquo;ll just include them +below without comment:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-line-to</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_line_to</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-width</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Numeric_Types.html#%28def._%28%28quote._~23~25foreign%29.__double%29%29">_double</a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_width</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-stroke</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_stroke</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The <span class="stt">cairo_set_line_cap</span> case is more interesting because the type +<span class="stt">cairo_line_cap_t</span> is a C enumeration type. Racket&rsquo;s FFI comes with +convenience forms for defining enumeration types&#8212; + <wbr />though it&rsquo;s possible +to encode them yourself too. The general philosophy of the Racket FFI +is to keep the C parts to a minimum and let you build abstractions +in Racket libraries. Here&rsquo;s a quote from the Barzilay and Orlovsky +paper on that:</p> + +<blockquote class="SubFlow"> + <p>Our design follows a simple principle: keep C-level +functionality to a minimum.</p></blockquote> + +<p>and specifically about enumerations:</p> + +<blockquote class="SubFlow"> + <p>For example, the C level part of our interface does not commit to a +specific implementation for enumerations &#8212; it simply exposes C integers.</p></blockquote> + +<p>To define an enumeration, we can use the <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> form. This procedure +sets up a new C type which converts between Racket symbols and the underlying +integer representations. For the <span class="stt">cairo_line_cap_t</span> type, it suffices to +just supply the cases as a list of symbols:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/define.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define%29%29">define</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">(</span><span class="RktVal">butt</span><span class="hspace">&nbsp;</span><span class="RktVal">round</span><span class="hspace">&nbsp;</span><span class="RktVal">square</span><span class="RktVal">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<p>The exact symbols that we specify are not important, since they just map to +integers anyway. The choice depends on what is convenient for the Racket interface. +It&rsquo;s also possible to specify how the symbols map to integers more precisely +(see the docs on <span class="RktWrap"><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Enumerations_and_Masks.html#%28def._%28%28lib._ffi%2Funsafe..rkt%29.__enum%29%29">_enum</a></span></span> for those details).</p> + +<p>Given this type, we can specify the type for the line cap function:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">define-cairo</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo-set-line-cap</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29.__fun%29%29">_fun</a></span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_t</span><span class="hspace">&nbsp;</span><span class="RktSym">_cairo_line_cap_t</span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/foreign_procedures.html#%28form._%28%28lib._ffi%2Funsafe..rkt%29._-~3e%29%29"><span class="nobreak">-&gt;</span></a></span><span class="hspace">&nbsp;</span><span class="RktSym"><a class="RktValLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/foreign/Other_Atomic_Types.html#%28def._%28%28quote._~23~25foreign%29.__void%29%29">_void</a></span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;&nbsp;</span><span class="hspace">&nbsp;&nbsp;</span><span class="RktPn">#:c-id</span><span class="hspace">&nbsp;</span><span class="RktSym">cairo_set_line_cap</span><span class="RktPn">)</span></td></tr></tbody></table></td></tr></tbody></table></div> + +<h1><a name="(part._.Putting_it_all_together)"></a>Putting it all together</h1> + +<p>Now that we have foreign function definitions for all of the relevant procedures, +we can just transcribe the example from the beginning into Racket syntax:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">75.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">125.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-move-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">50.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-line-to</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">200.0</span><span class="hspace">&nbsp;</span><span class="RktVal">175.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="hspace">&nbsp;</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-width</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">30.0</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-set-line-cap</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="hspace">&nbsp;</span><span class="RktVal">'</span><span class="RktVal">round</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="RktPn">(</span><span class="RktSym">cairo-stroke</span><span class="hspace">&nbsp;</span><span class="RktSym">ctx</span><span class="RktPn">)</span></td></tr></tbody></table></div> + +<p>Executing these procedure calls will draw into the Cairo surface we set up earlier, +which is connected to our original Racket bitmap object <span class="RktWrap"><span class="RktSym">bt</span></span>. To see the +results of what we drew, we can just evaluate <span class="RktWrap"><span class="RktSym">bt</span></span> at the REPL. But it&rsquo;s +a little nicer if we use the <span class="RktWrap"><a class="RktModLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/pict/index.html"><span class="RktSym">pict</span></a></span> library to draw a frame around +it to distinguish it from the background:</p> + +<div class="SCodeFlow"> + <table cellpadding="0" cellspacing="0" class="RktBlk"> + <tbody> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym"><a class="RktStxLink Sq" data-pltdoc="x" href="https://docs.racket-lang.org/reference/require.html#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._require%29%29">require</a></span><span class="hspace">&nbsp;</span><span class="RktSym">pict</span><span class="RktPn">)</span></td></tr> + <tr> + <td><span class="stt">&gt; </span><span class="RktPn">(</span><span class="RktSym">linewidth</span><span class="hspace">&nbsp;</span><span class="RktVal">2</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">frame</span><span class="hspace">&nbsp;</span><span class="RktPn">(</span><span class="RktSym">bitmap</span><span class="hspace">&nbsp;</span><span class="RktSym">bt</span><span class="RktPn">)</span><span class="RktPn">)</span><span class="RktPn">)</span></td></tr> + <tr> + <td> + <p><img src="/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png" alt="image" height="262" style="vertical-align: 0px; margin: -3px -3px -3px -3px;" width="262" /></p></td></tr></tbody></table></div> + +<p>And we&rsquo;re done! Of course, there is a lot more to the FFI. For example, I haven&rsquo;t +covered how to handle C functions that return multiple results through pointer +arguments. Or how to interoperate between Racket and C structs. I&rsquo;m hoping to +cover these in a future blog post, but in the meantime happy FFI hacking!</p> \ No newline at end of file diff --git a/blog/feeds/tutorials.atom.xml b/blog/feeds/tutorials.atom.xml new file mode 100644 index 00000000..e35e52ba --- /dev/null +++ b/blog/feeds/tutorials.atom.xml @@ -0,0 +1,16 @@ + + + PRL Blog: Posts tagged 'tutorials' + + + urn:http-prl-ccs-neu-edu:-blog-tags-tutorials-html + 2018-04-12T12:12:53Z + + [Making an IDE Plugin for DrRacket (cross-post)](https://lang.video/blog/2018/03/21/making-an-ide-plugin-for-drracket/) + + urn:http-prl-ccs-neu-edu:-blog-2018-04-12-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket + 2018-04-12T12:12:53Z + 2018-04-12T12:12:53Z + + Leif Andersen + \ No newline at end of file diff --git a/blog/feeds/tutorials.rss.xml b/blog/feeds/tutorials.rss.xml new file mode 100644 index 00000000..8ad91521 --- /dev/null +++ b/blog/feeds/tutorials.rss.xml @@ -0,0 +1,16 @@ + + + + PRL Blog: Posts tagged 'tutorials' + PRL Blog: Posts tagged 'tutorials' + http://prl.ccs.neu.edu/blog/tags/tutorials.html + Thu, 12 Apr 2018 12:12:53 UT + Thu, 12 Apr 2018 12:12:53 UT + 1800 + + [Making an IDE Plugin for DrRacket (cross-post)](https://lang.video/blog/2018/03/21/making-an-ide-plugin-for-drracket/) + http://prl.ccs.neu.edu/blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/?utm_source=tutorials&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2018-04-12-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket + Thu, 12 Apr 2018 12:12:53 UT + Leif Andersen + \ No newline at end of file diff --git a/blog/feeds/typed-racket.atom.xml b/blog/feeds/typed-racket.atom.xml new file mode 100644 index 00000000..7a7250e9 --- /dev/null +++ b/blog/feeds/typed-racket.atom.xml @@ -0,0 +1,1113 @@ + + + PRL Blog: Posts tagged 'typed racket' + + + urn:http-prl-ccs-neu-edu:-blog-tags-typed-racket-html + 2020-11-12T10:15:16Z + + Transient for Optional and Keyword Functions + + urn:http-prl-ccs-neu-edu:-blog-2020-11-12-transient-for-optional-and-keyword-functions + 2020-11-12T10:15:16Z + 2020-11-12T10:15:16Z + + Ben Greenman + +<p>A short adventure into the depths of optional and/or keyword functions in Racket.</p> +<!-- more--> + +<hr /> + +<p>Transient, or rather <em>the Transient semantics for a mixed-typed language</em>, is one way to let statically-typed code safely interact with untyped code. You can read all about it in <a href="http://hdl.handle.net/2022/23172">Michael Vitousek&rsquo;s 2019 dissertation</a> or <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my 2020 dissertation</a>, and you can see how it compares to other mixed-typed semantics <a href="http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">here</a>. The idea is to give up on <a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">behavioral type guarantees</a> and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called <em>shape checks</em>; for example:</p> + +<ul> + <li>if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;</li> + <li>if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and</li> + <li>if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.</li></ul> + +<p>Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type <strong>T</strong> into a shape check, and we have a function with type <strong>T</strong> that needs to check its inputs. The question is how to actually do the check in Racket v7.9.</p> + +<p>In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if <strong>T = (-&gt; Symbol Boolean)</strong> then we need to check the shape <strong>symbol?</strong> of the domain type <strong>Symbol</strong>:</p> + +<pre><code>;; source code +(: f (-&gt; Symbol Boolean)) +(define (f sym) + (eq? sym 'hola)) + +;; ===&gt; + +;; imaginary (but realistic) rewritten code +(define (f sym) + (assert sym symbol?) + (eq? sym 'hola))</code></pre> + +<p>A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type <strong>T</strong> accepts 1 to 3 inputs:</p> + +<pre><code>;; source code +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) +(define (g #:a a [b 'b] #:c [c #f]) + (if a b (if c 'left 'right))) + +;; ===&gt; + +;; imaginary, unrealistic rewritten code +(define (g #:a a [b 'b] #:c [c #f]) + (assert a boolean?) + (assert b symbol?) + (assert c void?) + (if a b (if c 'left 'right)))</code></pre> + +<p>Good &mdash; we basically know what we want. If the Racket core language had optional and keyword functions, then we&rsquo;d be done.</p> + +<p>But no, Racket expands these optional/keyword functions into primitive <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._lam))"><strong>lambda</strong></a> and <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._case-lam))"><strong>case-lambda</strong></a> forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.</p> + +<p>Let&rsquo;s keep digging.</p> + +<p>From now on, &ldquo;Shallow&rdquo; or &ldquo;Shallow TR&rdquo; refers to my implementation of Transient for Typed Racket (TR). We&rsquo;ll talk about Shallow instead of &ldquo;Transient&rdquo; in case future work reveals a better way to implement the Transient idea.</p> + +<h2 id="false-start-follow-the-type">False Start: Follow the Type</h2> + +<p>Beware &mdash; Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function <strong>g</strong> above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional <strong>#:c</strong> keyword lead to a shape-check failure because the variable <strong>c</strong> gets the default value <strong>#f</strong> instead of a void value. Concretely, the third assert from above fails:</p> + +<pre><code>(define (g #:a a [b 'b] #:c [c #f]) + .... + (assert c void?) ;; fails if c is the #f default value + ....)</code></pre> + +<p>The problem arises from subtyping. According to the annotations, the function <strong>g</strong> has an external type that is less precise than the internal type that validates the function body:</p> + +<pre><code>;; external type T +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) + +;; internal type T2, subtype of external (T2 &lt;: T), validates body +(: g (-&gt;* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))</code></pre> + +<p>Thanks to this external / internal distinction, the following easy rewrite idea, <em>Solution 0</em>, fails. Despite the failure, this first solution is a useful starting point for a success.</p> + +<h4 id="solution-0-step-1-mimic-the-typechecker">Solution 0, Step 1: Mimic the Typechecker</h4> + +<p>Shallow TR uses the same type checker as classic <em>Deep</em> TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.</p> + +<p>Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.</p> + +<p>Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (<a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-expr-unit.rkt#L274-L295">typecheck/tc-expr-unit.rkt</a>). The omitted code (<strong>&hellip;.</strong>) does actual type checking:</p> + +<pre><code>(define (tc-expr/check/internal form expected-type) + .... + (syntax-parse form + #:literal-sets (kernel-literals tc-expr-literals) + .... + [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^) + ....] + [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>Ok! Those two patterns say a lot about the expansion of optional and keyword functions:</p> + +<ol> + <li>Both forms expand to a <strong>let-values</strong> that binds one function <strong>fun</strong>.</li> + <li>TR uses the syntax classes <strong>kw-lambda^</strong> and <strong>opt-lambda^</strong> to tell these particular <strong>let-values</strong> apart from others.</li></ol> + +<p>Shallow TR can use exactly these patterns to recognize optional/keyword functions.</p> + +<h4 id="solution-0-step-2-parse-the-domain-type">Solution 0, Step 2: Parse the Domain Type</h4> + +<p>Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function&rsquo;s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:</p> + +<pre><code> [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/type-table.rkt#L80"><strong>type-of</strong></a> function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, <strong>fun</strong>, has a type. Great.</p> + +<p>Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:</p> + +<ul> + <li>The type of <strong>fun</strong> could be a straightforward <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L693"><strong>Fun type</strong></a>, but it could also be a: <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L701"><strong>DepFun type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L521"><strong>Poly type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L531"><strong>PolyDots type</strong></a>, or even a <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L900"><strong>Union type</strong></a>.</li> + <li>Each part of the domain type corresponds to one parameter of the <strong>fun</strong> expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in <strong>fun</strong>, or the mandatory keywords?</li></ul> + +<h4 id="solution-0-step-3-insert-a-shape-check">Solution 0, Step 3: Insert a Shape Check</h4> + +<p>Once we have the target <strong>fun</strong> expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse <strong>fun</strong> to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting <strong>fun</strong> goes something like this:</p> + +<pre><code>(syntax-parse fun + [(#%plain-lambda formals . body) + #:with (shape-check ...) + (make-shape-checks #'formals (type-of fun)) + #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])</code></pre> + +<p>The rewritten function executes shape checks immediately, and then proceeds with the <strong>body</strong> after validating each actual parameter.</p> + +<h2 id="on-the-trail-optkey-expansion">On the Trail: optkey Expansion</h2> + +<p>Our <em>Solution 0</em> fails because the type of the <strong>fun</strong> expression that it gets from the type-checked code is an external type. In terms of the <strong>g</strong> function from above, <em>Solution 0</em> uses the type <strong>Void</strong> instead of the internal type <strong>(U Void #f)</strong> to check the <strong>c</strong> parameter. To get internal types, we need to look closer at <strong>fun</strong> and the rest of the optional/keyword expansion.</p> + +<p>Let&rsquo;s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.</p> + +<p>If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function <strong>X</strong> create a module <strong>test.rkt</strong> like this:</p> + +<pre><code>#lang racket/base + +(define _ignore + (let () + X + (void)))</code></pre> + +<p>Invoke the expander with <code>raco expand test.rkt &gt; test.rkt.txt</code> and explore the generated <strong>.txt</strong> file.</p> + +<h3 id="example-1-mandatory-keyword">Example 1: mandatory keyword</h3> + +<p>The source is a function with one mandatory positional argument and one optional positional argument.</p> + +<pre><code>(lambda (x [y 0]) + (+ x y))</code></pre> + +<p>Expansion generates a <strong>case-lambda</strong> that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function <strong>F</strong> that expects two arguments, resolves defaults in a different way, and executes the function body.</p> + +<pre><code>(let-values (((F) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F x '0)) + ((x y1) (#%app F x y1))))</code></pre> + +<p>Note: the expression <strong>(if &rsquo;#f &rsquo;0 y1)</strong> in the generated <strong>F</strong> function is equal to <strong>y1</strong> alone. In general, the <strong>if</strong> is for default expressions. (<a href="https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/">Unlike Python</a>, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a <strong>#f</strong> test. A general-purpose optimizer can remove this test before the code runs.</p> + +<h3 id="example-2">Example 2:</h3> + +<p>The source is a function with one mandatory positional argument and one mandatory keyword argument:</p> + +<pre><code>(lambda (x #:y y) + (+ x y))</code></pre> + +<p>Expansion generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> expects a list of keywords, a list of arguments, and a final argument. The purpose of <strong>F1</strong> is to organize a call to <strong>F0</strong>.</li> + <li><strong>lifted/2</strong> is the constructor for a generated struct type. Other functions help the struct call <strong>F1</strong>. Nevermind the details; I don&rsquo;t fully understand them either.</li></ul> + +<p>The important piece for Shallow TR is the <strong>F0</strong> function because the goal of rewriting is to protect the original function body against untyped inputs.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) y1)) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y1) (#%app car given-args))) + (#%app F0 y1 x3))))) + (#%app + lifted/2 + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l2571) given-kws)) + (if (#%app pair? l2571) + (if (#%app eq? (#%app car l2571) '#:y) + (#%app null? (#%app cdr l2571)) + '#f) + '#f)) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + '(#:y) + '(#:y))))</code></pre> + +<h3 id="example-3">Example 3:</h3> + +<p>The source is a function with one mandatory positional argument and one optional keyword argument:</p> + +<pre><code>(lambda (x #:y [y 0]) + (+ x y))</code></pre> + +<p>Expansion again generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments, resolves the optional default, and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> calls <strong>F0</strong></li> + <li>At the bottom, there are two <strong>case-lambda</strong> functions that call <strong>F1</strong></li></ul> + +<p>Again, the <strong>F0</strong> function is the focal point for Shallow TR rewriting.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y2) (#%app pair? given-kws))) + (let-values (((y1) + (if y2 (#%app car given-args) '0))) + (#%app F0 y1 x3)))))) + (#%app + make-optional-keyword-procedure + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l1571) given-kws)) + (let-values (((l1571) + (if (#%app null? l1571) + l1571 + (if (#%app eq? (#%app car l1571) '#:y) + (#%app cdr l1571) + l1571)))) + (#%app null? l1571))) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + null + '(#:y) + (case-lambda + ((x) (#%app F1 null null x))))))</code></pre> + +<h2 id="solution-the-shallow-tr-rewrite-strategy">Solution: The Shallow TR Rewrite Strategy</h2> + +<p>All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a <strong>let-values</strong> form:</p> + +<pre><code>(let-values (((f) fun)) . body)</code></pre> + +<p>Furthermore, the generated <strong>fun</strong> is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the <strong>fun</strong> from <em>Example 3</em> again; it has formal parameters for the keyword arg. and the mandatory arg., and one <strong>let-values</strong> to resolve each parameter:</p> + +<pre><code> (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y)))))</code></pre> + +<p>Another experiment with <strong>type-of</strong> shows that the right-hand side of each <strong>let-values</strong> has an internal type annotation. Excellent! Both <strong>(type-of x3)</strong> and <strong>(type-of (if &rsquo;#f &rsquo;0 y1))</strong> are the right types for shape checks. Shallow TR can:</p> + +<ul> + <li>inspect the <strong>let-values</strong> one-by-one;</li> + <li>convert the type of each right-hand expression to a shape predicate; and</li> + <li>rewrite each right-hand <strong>expr</strong> into <strong>(assert expr shape?)</strong>.</li></ul> + +<p>This should work! In fact, we can do slightly better:</p> + +<ul> + <li>when the right-hand expression is a conditional <strong>(if test default-expr supplied-arg)</strong></li> + <li>then Shallow only needs to check the supplied arg: <strong>(if test default-expr (assert supplied-arg shape?))</strong></li></ul> + +<p>Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.</p> + +<h2 id="a-problem-with-methods-and-a-bugfix">A Problem with Methods and a Bugfix</h2> + +<p>Currently, Shallow TR rewrites optional and keyword functions using the <strong>let-values</strong> plan described above. Each formal parameter has one <strong>let-values</strong> binding, and the type on each bound expression defines the shape check.</p> + +<p>Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we <a href="https://github.com/racket/racket/pull/3182">fixed the class expander</a>.</p> + +<p>First, here is a class with one method that runs correctly. The method <strong>f</strong> accepts an optional positional argument <strong>x</strong>; the default value of <strong>x</strong> is the current value of the field <strong>my-num</strong> (fields are mutable):</p> + +<pre><code>(define c0% + (class object% + (super-new) + (field (my-num 2)) + (define/public (f [x my-num]) + (+ x x))))</code></pre> + +<p>Second, here is a similar method that fails. This time, the default is an immediate value <strong>2</strong>:</p> + +<pre><code>(define c1% + (class object% + (super-new) + (define/public (f [x 2]) + (+ x x))))</code></pre> + +<p>Running a call <strong>(send o1 f)</strong> used to raise a shape-check failure about a strange value:</p> + +<blockquote> + <p>shape error: Expected a real number, got <code>#&lt;unsafe-undefined&gt;</code></p></blockquote> + +<p>What is going on?</p> + +<p>It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:</p> + +<pre><code>(lambda (x [y z]) + (+ x y))</code></pre> + +<p>Expansion generates a function <strong>F0</strong> that checks for the undefined value, and an outer <strong>case-lambda</strong> that supplies undefined when the default is needed:</p> + +<pre><code>(let-values (((F0) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) + (if (#%app eq? y1 unsafe-undefined) + z + y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F0 x unsafe-undefined)) + ((x y1) (#%app F0 x y1))))</code></pre> + +<p>That&rsquo;s the normal way that <strong>unsafe-undefined</strong> shows up: the <a href="https://github.com/racket/racket/blob/c0ff11e27bd28e070c20b7a9b0f7365f8f2b665a/racket/collects/racket/private/kw.rkt">expander for optional/keyword functions</a> looks for default expressions vs. default values and uses the undefined value for expressions.</p> + +<p>Three other facts conspired to make the problem with optional methods:</p> + +<ol> + <li>Typed Racket also looks for default expressions vs. default values (search for <strong>immediate-default</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/base-env/annotate-classes.rkt">here</a>). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the <strong>unsafe-undefined</strong> value (search for <strong>-Unsafe-Undefined</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/kw-types.rkt">here (kw)</a> and <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-lambda-unit.rkt">here (opt)</a>).</li> + <li>The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.</li> + <li>Shallow TR pushes default expression checks <strong>(if test default-expr supplied-arg)</strong> to the <strong>supplied-arg</strong> instead of wrapping the whole <strong>if</strong> form.</li></ol> + +<p>In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound&mdash;but harmless because the false assumption was guarded by an <strong>if</strong> test for <strong>unsafe-undefined</strong>. Running Shallow TR revealed the unsoundness with its eager shape check.</p> + +<p>Again, the resolution was to fix the class expander (<a href="https://github.com/racket/racket/pull/3182">racket/racket #3182</a>). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.</p> + +<h2 id="lessons">Lessons</h2> + +<ol> + <li>Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.</li> + <li>Digging into the expansion is sometimes necessary. There are at least three places that do so&mdash;the class expander, TR, and Shallow TR&mdash;and unfortunately they all need to cooperate.</li> + <li>The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my dissertation</a> lists them all.</li></ol> + + Transient Answers Old Questions + + urn:http-prl-ccs-neu-edu:-blog-2020-10-15-transient-answers-old-questions + 2020-10-15T13:32:12Z + 2020-10-15T13:32:12Z + + Ben Greenman + +<p>Several old questions from the Typed Racket mailing list have new and simple answers under a &ldquo;transient&rdquo; Typed Racket.</p> +<!-- more--> + +<hr /> + +<p>For the past few months, I&rsquo;ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the <a href="https://github.com/racket/typed-racket/pull/952">RFC</a> and <a href="https://github.com/racket/typed-racket/pull/948">pull request</a>.</p> + +<p>The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that&rsquo;s about it. By contrast, types are much stronger in classic Typed Racket.</p> + +<p>Shallow Racket&rsquo;s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the <a href="https://groups.google.com/g/racket-users">Racket mailing list</a>. There were a surprising number of questions that went like this:</p> + +<blockquote> + <p><strong>Q.</strong> Hey, I ran a program expecting <em>X</em> to happen, but <em>Y</em> happened instead. Is this a bug?</p> + <p><strong>A.</strong> No, Typed Racket has to do <em>Y</em> because of its strong types.</p></blockquote> + +<p>&hellip; but changing to shallow types gives the <em>X</em> behavior! Here are their stories.</p> + +<p>Going forward, <strong>Deep</strong> refers to normal Typed Racket and <strong>Shallow</strong> refers to Shallow Typed Racket.</p> + +<hr /> + +<h2 id="higher-order-value-as-any">Higher-Order Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ">groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ</a></p> + +<h4 id="on-20180416-mailoo-wrote">On 2018&ndash;04&ndash;16, <em>mailoo</em> wrote:</h4> + +<blockquote> + <p> I play a little with the &ldquo;Any&rdquo; type (due to &lsquo;dynamic-require&rsquo; which return Any), and I&rsquo;m not able to cast them back in a function.</p> + <p> I (over) simplify my question with this little program :</p></blockquote> + +<pre><code>(: p Any) +(define (p i) (displayln i)) + +; Here I want to get back my function +(define proc (cast p (-&gt; Integer Void))) +(proc 2) </code></pre> + +<blockquote> + <p> but I get this error when I try to execute the function :</p></blockquote> + +<pre><code>; contract violation +; Attempted to use a higher-order value passed as `Any` in untyped code: #&lt;procedure:p&gt; </code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> raises an error because it must enforce the <code>Any</code> type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> prints 2 and returns void. No error. Same goes for dynamic-require.</p> + +<hr /> + +<h2 id="parametric-contract-affects-untyped-code">Parametric Contract Affects Untyped Code</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ">groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ</a></p> + +<h4 id="on-20191215-john-clements-wrote">On 2019&ndash;12&ndash;15, John Clements wrote:</h4> + +<blockquote> + <p> It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:</p></blockquote> + +<pre><code> #lang typed/racket + + (require/typed racket/list + [index-of (All (T) ((Listof T) T -&gt; (U False Natural)))]) + + (index-of '(n s e w) 'n) ;; returns... #f? </code></pre> + +<blockquote> + <p> In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).</p> + <p> I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces parametricity for <code>All</code> types, and this throws off the equality function that index-of uses internally.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> returns 0.</p> + +<p>ps John, thanks very much for working on <a href="https://adventofcode.com">Advent of Code</a> and mailing the list!</p> + +<hr /> + +<h2 id="unable-to-protect-opaque-value-as-any">Unable to Protect Opaque Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ">groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ</a></p> + +<h4 id="on-20191211-marc-kaufmann-wrote">On 2019&ndash;12&ndash;11, Marc Kaufmann wrote:</h4> + +<blockquote> + <p>I have one file called <code>type-test.rkt</code> with the following</p></blockquote> + +<pre><code>#lang typed/racket + +(require (only-in typed/web-server/http response/xexpr response)) + +(provide f2) + +(: f2 (-&gt; (U response Any))) +(define (f2) + (define x '(body (h1 "Try it"))) + (: resp response) + (define resp (response/xexpr x)) + resp)</code></pre> + +<blockquote> + <p>Then I have another <em>untyped</em> file for a servlet:</p></blockquote> + +<pre><code>#lang racket + +(require "type-test.rkt" + web-server/servlet + web-server/servlet-env) + +(define (start req) + (f2)) + +(serve/servlet start + #:servlet-regexp #rx"" + #:launch-browser? #false + #:port 8080)</code></pre> + +<blockquote> + <p>Notice that I am telling [f2] that <code>resp</code> is of type <code>response</code>. Yet, when I run the server with <code>start</code> [&hellip;.] I get the following result:</p> + <p>(f2): Error, see below.</p> + <p>The error is:</p></blockquote> + +<pre><code>f2: broke its own contract + any-wrap/c: Unable to protect opaque value passed as `Any` + value: #&lt;response&gt; + in: the range of + (-&gt; Any)</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the <code>Any</code> type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.</p> + +<hr /> + +<h2 id="type-inference-installs-a-precise-type">Type Inference Installs a Precise Type</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ">groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ</a></p> + +<h4 id="on-20200214-john-clements-wrote">On 2020&ndash;02&ndash;14, John Clements wrote:</h4> + +<blockquote> + <p>I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.</p> + <p>Here’s a program:</p></blockquote> + +<pre><code>#lang typed/racket + +(define-type Store (Mutable-HashTable Integer Value)) +(define-type Value (U Real Boolean String)) + +(define top-store + (cast + (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f))) + Store)) + +(hash-set! top-store 5 1234)</code></pre> + +<blockquote> + <p>It fails with this error:</p></blockquote> + +<pre><code>contract violation +expected: (or/c (and/c byte? positive?) #t #f) +given: 1234 +in: the values of +the 3rd conjunct of +(and/c hash? + hash-mutable? + (hash/c exact-integer? + (or/c (and/c byte? positive?) #t #f) + #:immutable #f))</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p>First off, <strong>Deep</strong> runs fine after swapping <code>cast</code> for <code>ann</code>.</p> + +<p>Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.</p> + +<p>The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.</p> + +<hr /> + +<h2 id="same-arity-functions-in-a-case-lambda">Same-Arity Functions in a Case Lambda</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ">groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ</a></p> + +<h4 id="on-20190705-ryan-kramer-wrote">On 2019&ndash;07&ndash;05, Ryan Kramer wrote:</h4> + +<blockquote> + <p>In the code below, can <code>maybe-car</code> have the given type [&hellip;.]?</p></blockquote> + +<pre><code>#lang typed/racket + +(module untyped racket + (provide maybe-car) + (define (maybe-car x) + (cond + [(pair? x) (car x)] + [else x]))) + +(require/typed + 'untyped + [maybe-car (All (a b) (case-&gt; + (-&gt; (Pairof a b) a) + (-&gt; a a)))])</code></pre> + +<blockquote> + <p>[Current error:]</p></blockquote> + +<pre><code>Type Checker: + Type (All (a b) (case-&gt; (-&gt; (Pairof a b) a) (-&gt; a a))) + could not be converted to a contract: + function type has two cases of arity 1</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the type with a Racket <code>or/c</code> contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don&rsquo;t know which to apply to an incoming function.</p> + +<p>Note, the &ldquo;Type Checker&rdquo; error message is much better than what or/c would give on its own.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> simply checks that maybe-car accepts both arities inside the case-&gt; type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.</p> + +<hr /> + +<h3 id="immutable-type-affects-untyped-code">Immutable Type Affects Untyped Code</h3> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ">groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ</a></p> + +<h4 id="on-20200217-bertrand-augereau-wrote">On 2020&ndash;02&ndash;17, Bertrand Augereau wrote:</h4> + +<blockquote> + <p>Hello everybody, I&rsquo;m trying to gradually type my script to make it a proper app (yes I&rsquo;m a static-ish guy) and I have an issue (Racket 7.6 CS).</p></blockquote> + +<pre><code>; racket_mod.rkt: +#lang racket + +(provide (struct-out s)) +(provide list-of-s) +(provide set-list-of-s!) + +(struct s (a)) +(define list-of-s '()) +(define (set-list-of-s! los) + (set! list-of-s los))</code></pre> + +<pre><code>; racket_mod_typed.rkt: +#lang typed/racket + +(provide (struct-out s2)) +(provide list-of-s2) +(provide set-list-of-s2!) + +(struct s2 ([a : Natural])) +(define list-of-s2 '()) +(define (set-list-of-s2! [los : (Listof s2)]) + (set! list-of-s2 los))</code></pre> + +<pre><code>; racket_main.rkt: +#lang racket + +(require "racket_mod.rkt") +(require "racket_mod_typed.rkt") + +(define los (list (s 1) (s 2))) +(set-list-of-s! los) +(displayln list-of-s) + +(define los2 (list (s2 1) (s2 2))) +(set-list-of-s2! los2) +(displayln list-of-s2)</code></pre> + +<blockquote> + <p>list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces the type of <code>list-of-s2</code> with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.</p> + +<p>Here&rsquo;s a step-by-step breakdown:</p> + +<ol> + <li>the typed module creates an empty list-of-s2</li> + <li>the main module imports the list and receives a new copy</li> + <li>the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable</li> + <li>the main module reads from its copy &mdash; and it&rsquo;s still empty</li></ol> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> lets the original list travel to untyped code. There are no contracts in the way.</p> + +<h2 id="discussion">Discussion</h2> + +<p>Wow! It&rsquo;s great to see that Shallow Racket works &ldquo;as expected&rdquo; on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.</p> + +<p>If you have a similar experience with a deep-types error, let me know.</p> + +<p>Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error &mdash; and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.</p> + +<p>Shallow Typed Racket is coming soon. Follow the <a href="https://github.com/racket/typed-racket/pull/948">pull request</a> or watch the Racket release notes for news.</p> + +<h3 id="links">Links</h3> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">Larger example</a> where Shallow misses an error that Deep catches</li> + <li>Michael M. Vitousek <a href="http://hdl.handle.net/2022/23172">invented</a> the Transient semantics and implemented it in <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>.</li> + <li>My <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-thesis-2020">upcoming dissertation</a> has lots more to say about Shallow Typed Racket.</li></ul> + +<p><em>Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.</em></p> + + The Typed Racket Optimizer vs. Transient + + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + 2020-01-15T12:16:35Z + 2020-01-15T12:16:35Z + + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> \ No newline at end of file diff --git a/blog/feeds/typed-racket.rss.xml b/blog/feeds/typed-racket.rss.xml new file mode 100644 index 00000000..ee3c4812 --- /dev/null +++ b/blog/feeds/typed-racket.rss.xml @@ -0,0 +1,1109 @@ + + + + PRL Blog: Posts tagged 'typed racket' + PRL Blog: Posts tagged 'typed racket' + http://prl.ccs.neu.edu/blog/tags/typed-racket.html + Thu, 12 Nov 2020 10:15:16 UT + Thu, 12 Nov 2020 10:15:16 UT + 1800 + + Transient for Optional and Keyword Functions + http://prl.ccs.neu.edu/blog/2020/11/12/transient-for-optional-and-keyword-functions/?utm_source=typed-racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-11-12-transient-for-optional-and-keyword-functions + Thu, 12 Nov 2020 10:15:16 UT + Ben Greenman + +<p>A short adventure into the depths of optional and/or keyword functions in Racket.</p> +<!-- more--> + +<hr /> + +<p>Transient, or rather <em>the Transient semantics for a mixed-typed language</em>, is one way to let statically-typed code safely interact with untyped code. You can read all about it in <a href="http://hdl.handle.net/2022/23172">Michael Vitousek&rsquo;s 2019 dissertation</a> or <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my 2020 dissertation</a>, and you can see how it compares to other mixed-typed semantics <a href="http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/">here</a>. The idea is to give up on <a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">behavioral type guarantees</a> and focus on a (weak) form of type soundness. To enforce soundness, Transient rewrites every expression in typed code with assertions called <em>shape checks</em>; for example:</p> + +<ul> + <li>if a typed module imports an untyped library, then every value that crosses the module boundary gets a shape check;</li> + <li>if typed code reads from an array, then every element that comes out of the array must satisfy a shape check; and</li> + <li>if a typed function escapes to untyped code, then the function must use a shape check to validate every input that it receives.</li></ul> + +<p>Our goal today is to understand the shape checks for functions. Suppose we know how to turn a type <strong>T</strong> into a shape check, and we have a function with type <strong>T</strong> that needs to check its inputs. The question is how to actually do the check in Racket v7.9.</p> + +<p>In your standard theory, rewriting is no problem. A (simplified, model) function takes exactly one argument and needs exactly one shape check in the body; if <strong>T = (-&gt; Symbol Boolean)</strong> then we need to check the shape <strong>symbol?</strong> of the domain type <strong>Symbol</strong>:</p> + +<pre><code>;; source code +(: f (-&gt; Symbol Boolean)) +(define (f sym) + (eq? sym 'hola)) + +;; ===&gt; + +;; imaginary (but realistic) rewritten code +(define (f sym) + (assert sym symbol?) + (eq? sym 'hola))</code></pre> + +<p>A Typed Racket function can accept optional arguments, keyword arguments, and optional keyword arguments. These are still fairly easy to handle in theory. Below, the function type <strong>T</strong> accepts 1 to 3 inputs:</p> + +<pre><code>;; source code +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) +(define (g #:a a [b 'b] #:c [c #f]) + (if a b (if c 'left 'right))) + +;; ===&gt; + +;; imaginary, unrealistic rewritten code +(define (g #:a a [b 'b] #:c [c #f]) + (assert a boolean?) + (assert b symbol?) + (assert c void?) + (if a b (if c 'left 'right)))</code></pre> + +<p>Good &mdash; we basically know what we want. If the Racket core language had optional and keyword functions, then we&rsquo;d be done.</p> + +<p>But no, Racket expands these optional/keyword functions into primitive <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._lam))"><strong>lambda</strong></a> and <a href="https://docs.racket-lang.org/raco/decompile.html#(def._((lib._compiler%2Fzo-structs..rkt)._case-lam))"><strong>case-lambda</strong></a> forms. Typed Racket type-checks this expanded code, thus Shallow Typed Racket (the Transient version) must rewrite the expanded code.</p> + +<p>Let&rsquo;s keep digging.</p> + +<p>From now on, &ldquo;Shallow&rdquo; or &ldquo;Shallow TR&rdquo; refers to my implementation of Transient for Typed Racket (TR). We&rsquo;ll talk about Shallow instead of &ldquo;Transient&rdquo; in case future work reveals a better way to implement the Transient idea.</p> + +<h2 id="false-start-follow-the-type">False Start: Follow the Type</h2> + +<p>Beware &mdash; Shallow TR cannot rely on type annotations to decide which shape checks to insert. The example function <strong>g</strong> above demonstrates that annotations are not good enough. With our imagined rewrite, calls that leave out the optional <strong>#:c</strong> keyword lead to a shape-check failure because the variable <strong>c</strong> gets the default value <strong>#f</strong> instead of a void value. Concretely, the third assert from above fails:</p> + +<pre><code>(define (g #:a a [b 'b] #:c [c #f]) + .... + (assert c void?) ;; fails if c is the #f default value + ....)</code></pre> + +<p>The problem arises from subtyping. According to the annotations, the function <strong>g</strong> has an external type that is less precise than the internal type that validates the function body:</p> + +<pre><code>;; external type T +(: g (-&gt;* [#:a Boolean] [Symbol #:c Void] Symbol)) + +;; internal type T2, subtype of external (T2 &lt;: T), validates body +(: g (-&gt;* [#:a Boolean] [Symbol #:c (U #f Void)] Symbol))</code></pre> + +<p>Thanks to this external / internal distinction, the following easy rewrite idea, <em>Solution 0</em>, fails. Despite the failure, this first solution is a useful starting point for a success.</p> + +<h4 id="solution-0-step-1-mimic-the-typechecker">Solution 0, Step 1: Mimic the Typechecker</h4> + +<p>Shallow TR uses the same type checker as classic <em>Deep</em> TR. If type checking succeeds, then Shallow must insert shape checks. Otherwise, compilation stops with a type error.</p> + +<p>Thanks to its wholesale reuse of the type checker, Shallow TR can use syntax patterns from the type checker to navigate expanded Racket code. For optional and keyword functions in particular, Shallow can get started by looking at how the type checker recognizes these forms in expanded code.</p> + +<p>Here are two syntax patterns for keyword functions and optional functions in the Deep TR type checker (<a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-expr-unit.rkt#L274-L295">typecheck/tc-expr-unit.rkt</a>). The omitted code (<strong>&hellip;.</strong>) does actual type checking:</p> + +<pre><code>(define (tc-expr/check/internal form expected-type) + .... + (syntax-parse form + #:literal-sets (kernel-literals tc-expr-literals) + .... + [(~and (let-values ([(f) fun]) . body) kw:kw-lambda^) + ....] + [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>Ok! Those two patterns say a lot about the expansion of optional and keyword functions:</p> + +<ol> + <li>Both forms expand to a <strong>let-values</strong> that binds one function <strong>fun</strong>.</li> + <li>TR uses the syntax classes <strong>kw-lambda^</strong> and <strong>opt-lambda^</strong> to tell these particular <strong>let-values</strong> apart from others.</li></ol> + +<p>Shallow TR can use exactly these patterns to recognize optional/keyword functions.</p> + +<h4 id="solution-0-step-2-parse-the-domain-type">Solution 0, Step 2: Parse the Domain Type</h4> + +<p>Once the Shallow TR rewriter has found an optional/keyword function, the next step is to find the function&rsquo;s type and figure out the right shape check. For an optional function, the rewriter has an expression that matches the following pattern:</p> + +<pre><code> [(~and (let-values ([(f) fun]) . body) opt:opt-lambda^) + ....]</code></pre> + +<p>First, we need a type. The type checker decorates (almost) every expression with a type as a syntax property. (Unreachable code may not have a type.) The <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/type-table.rkt#L80"><strong>type-of</strong></a> function gets the type decoration from an expression. A little experimentation shows that the function part of our expression, <strong>fun</strong>, has a type. Great.</p> + +<p>Second, we need to parse the domain from the function type. This is easier said than done. Fortunately, our final solution does not need the parsing step so I will list the challenges and move on:</p> + +<ul> + <li>The type of <strong>fun</strong> could be a straightforward <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L693"><strong>Fun type</strong></a>, but it could also be a: <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L701"><strong>DepFun type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L521"><strong>Poly type</strong></a>, or <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L531"><strong>PolyDots type</strong></a>, or even a <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/rep/type-rep.rkt#L900"><strong>Union type</strong></a>.</li> + <li>Each part of the domain type corresponds to one parameter of the <strong>fun</strong> expression. Matching the parameter names to types is not straightforward; for example, do the mandatory parameters come first in <strong>fun</strong>, or the mandatory keywords?</li></ul> + +<h4 id="solution-0-step-3-insert-a-shape-check">Solution 0, Step 3: Insert a Shape Check</h4> + +<p>Once we have the target <strong>fun</strong> expression and a map from parameter names to types, the final step of our tentative solution is easy. First, convert the types to shape predicates. Second, parse <strong>fun</strong> to separate the parameters from the body. Third, insert a block of shape checks to the top of the body. All together, rewriting <strong>fun</strong> goes something like this:</p> + +<pre><code>(syntax-parse fun + [(#%plain-lambda formals . body) + #:with (shape-check ...) + (make-shape-checks #'formals (type-of fun)) + #'(#%plain-lambda formals (#%plain-app void shape-check ...) . body)])</code></pre> + +<p>The rewritten function executes shape checks immediately, and then proceeds with the <strong>body</strong> after validating each actual parameter.</p> + +<h2 id="on-the-trail-optkey-expansion">On the Trail: optkey Expansion</h2> + +<p>Our <em>Solution 0</em> fails because the type of the <strong>fun</strong> expression that it gets from the type-checked code is an external type. In terms of the <strong>g</strong> function from above, <em>Solution 0</em> uses the type <strong>Void</strong> instead of the internal type <strong>(U Void #f)</strong> to check the <strong>c</strong> parameter. To get internal types, we need to look closer at <strong>fun</strong> and the rest of the optional/keyword expansion.</p> + +<p>Let&rsquo;s study three example functions and their expanded forms. The expansions reveal a common pattern that motivates a new Shallow TR strategy.</p> + +<p>If you want to expand these examples yourself, hide them from the Racket toplevel as follows. For each example function <strong>X</strong> create a module <strong>test.rkt</strong> like this:</p> + +<pre><code>#lang racket/base + +(define _ignore + (let () + X + (void)))</code></pre> + +<p>Invoke the expander with <code>raco expand test.rkt &gt; test.rkt.txt</code> and explore the generated <strong>.txt</strong> file.</p> + +<h3 id="example-1-mandatory-keyword">Example 1: mandatory keyword</h3> + +<p>The source is a function with one mandatory positional argument and one optional positional argument.</p> + +<pre><code>(lambda (x [y 0]) + (+ x y))</code></pre> + +<p>Expansion generates a <strong>case-lambda</strong> that accepts one or two arguments. The one-argument case supplies a default value for the missing parameter. Both cases call a generated function <strong>F</strong> that expects two arguments, resolves defaults in a different way, and executes the function body.</p> + +<pre><code>(let-values (((F) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F x '0)) + ((x y1) (#%app F x y1))))</code></pre> + +<p>Note: the expression <strong>(if &rsquo;#f &rsquo;0 y1)</strong> in the generated <strong>F</strong> function is equal to <strong>y1</strong> alone. In general, the <strong>if</strong> is for default expressions. (<a href="https://pythonconquerstheuniverse.wordpress.com/2012/02/15/mutable-default-arguments/">Unlike Python</a>, Racket evaluates a mutable default once for each function call.) When the default is an immediate value, as this example illustrates, the expander generates a <strong>#f</strong> test. A general-purpose optimizer can remove this test before the code runs.</p> + +<h3 id="example-2">Example 2:</h3> + +<p>The source is a function with one mandatory positional argument and one mandatory keyword argument:</p> + +<pre><code>(lambda (x #:y y) + (+ x y))</code></pre> + +<p>Expansion generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> expects a list of keywords, a list of arguments, and a final argument. The purpose of <strong>F1</strong> is to organize a call to <strong>F0</strong>.</li> + <li><strong>lifted/2</strong> is the constructor for a generated struct type. Other functions help the struct call <strong>F1</strong>. Nevermind the details; I don&rsquo;t fully understand them either.</li></ul> + +<p>The important piece for Shallow TR is the <strong>F0</strong> function because the goal of rewriting is to protect the original function body against untyped inputs.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) y1)) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y1) (#%app car given-args))) + (#%app F0 y1 x3))))) + (#%app + lifted/2 + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l2571) given-kws)) + (if (#%app pair? l2571) + (if (#%app eq? (#%app car l2571) '#:y) + (#%app null? (#%app cdr l2571)) + '#f) + '#f)) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + '(#:y) + '(#:y))))</code></pre> + +<h3 id="example-3">Example 3:</h3> + +<p>The source is a function with one mandatory positional argument and one optional keyword argument:</p> + +<pre><code>(lambda (x #:y [y 0]) + (+ x y))</code></pre> + +<p>Expansion again generates several functions:</p> + +<ul> + <li><strong>F0</strong> expects a plain list of arguments, resolves the optional default, and executes the source function&rsquo;s body</li> + <li><strong>F1</strong> calls <strong>F0</strong></li> + <li>At the bottom, there are two <strong>case-lambda</strong> functions that call <strong>F1</strong></li></ul> + +<p>Again, the <strong>F0</strong> function is the focal point for Shallow TR rewriting.</p> + +<pre><code>(let-values (((F0) + (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y))))))) + (let-values (((F1) + (lambda (given-kws given-args x3) + (let-values (((y2) (#%app pair? given-kws))) + (let-values (((y1) + (if y2 (#%app car given-args) '0))) + (#%app F0 y1 x3)))))) + (#%app + make-optional-keyword-procedure + (lambda (given-kws given-argc) + (if (#%app = given-argc '3) + (let-values (((l1571) given-kws)) + (let-values (((l1571) + (if (#%app null? l1571) + l1571 + (if (#%app eq? (#%app car l1571) '#:y) + (#%app cdr l1571) + l1571)))) + (#%app null? l1571))) + '#f)) + (case-lambda + ((given-kws given-args x) + (#%app F1 given-kws given-args x))) + null + '(#:y) + (case-lambda + ((x) (#%app F1 null null x))))))</code></pre> + +<h2 id="solution-the-shallow-tr-rewrite-strategy">Solution: The Shallow TR Rewrite Strategy</h2> + +<p>All three examples show a common pattern among the expansions of optional and keyword functions. Each function expands to a <strong>let-values</strong> form:</p> + +<pre><code>(let-values (((f) fun)) . body)</code></pre> + +<p>Furthermore, the generated <strong>fun</strong> is a lambda that first resolves optional arguments and then executes the body of the original function. Here is the <strong>fun</strong> from <em>Example 3</em> again; it has formal parameters for the keyword arg. and the mandatory arg., and one <strong>let-values</strong> to resolve each parameter:</p> + +<pre><code> (lambda (y1 x3) + (let-values (((x) x3)) + (let-values (((y) (if '#f '0 y1))) + (let-values () (#%app + x y)))))</code></pre> + +<p>Another experiment with <strong>type-of</strong> shows that the right-hand side of each <strong>let-values</strong> has an internal type annotation. Excellent! Both <strong>(type-of x3)</strong> and <strong>(type-of (if &rsquo;#f &rsquo;0 y1))</strong> are the right types for shape checks. Shallow TR can:</p> + +<ul> + <li>inspect the <strong>let-values</strong> one-by-one;</li> + <li>convert the type of each right-hand expression to a shape predicate; and</li> + <li>rewrite each right-hand <strong>expr</strong> into <strong>(assert expr shape?)</strong>.</li></ul> + +<p>This should work! In fact, we can do slightly better:</p> + +<ul> + <li>when the right-hand expression is a conditional <strong>(if test default-expr supplied-arg)</strong></li> + <li>then Shallow only needs to check the supplied arg: <strong>(if test default-expr (assert supplied-arg shape?))</strong></li></ul> + +<p>Note: Shallow needs to rewrite the default expression, but it can trust its final shape because of (Transient) type soundness.</p> + +<h2 id="a-problem-with-methods-and-a-bugfix">A Problem with Methods and a Bugfix</h2> + +<p>Currently, Shallow TR rewrites optional and keyword functions using the <strong>let-values</strong> plan described above. Each formal parameter has one <strong>let-values</strong> binding, and the type on each bound expression defines the shape check.</p> + +<p>Last May, though, this rewriting caused new failures in methods with optional arguments. The failure was due to a mismatch between Typed Racket and the Racket class expander. Since then, we <a href="https://github.com/racket/racket/pull/3182">fixed the class expander</a>.</p> + +<p>First, here is a class with one method that runs correctly. The method <strong>f</strong> accepts an optional positional argument <strong>x</strong>; the default value of <strong>x</strong> is the current value of the field <strong>my-num</strong> (fields are mutable):</p> + +<pre><code>(define c0% + (class object% + (super-new) + (field (my-num 2)) + (define/public (f [x my-num]) + (+ x x))))</code></pre> + +<p>Second, here is a similar method that fails. This time, the default is an immediate value <strong>2</strong>:</p> + +<pre><code>(define c1% + (class object% + (super-new) + (define/public (f [x 2]) + (+ x x))))</code></pre> + +<p>Running a call <strong>(send o1 f)</strong> used to raise a shape-check failure about a strange value:</p> + +<blockquote> + <p>shape error: Expected a real number, got <code>#&lt;unsafe-undefined&gt;</code></p></blockquote> + +<p>What is going on?</p> + +<p>It turns out, the undefined value comes from the expander. Here is an optional function with a default expression:</p> + +<pre><code>(lambda (x [y z]) + (+ x y))</code></pre> + +<p>Expansion generates a function <strong>F0</strong> that checks for the undefined value, and an outer <strong>case-lambda</strong> that supplies undefined when the default is needed:</p> + +<pre><code>(let-values (((F0) + (lambda (x2 y1) + (let-values (((x) x2)) + (let-values (((y) + (if (#%app eq? y1 unsafe-undefined) + z + y1))) + (let-values () (#%app + x y))))))) + (case-lambda + ((x) (#%app F0 x unsafe-undefined)) + ((x y1) (#%app F0 x y1))))</code></pre> + +<p>That&rsquo;s the normal way that <strong>unsafe-undefined</strong> shows up: the <a href="https://github.com/racket/racket/blob/c0ff11e27bd28e070c20b7a9b0f7365f8f2b665a/racket/collects/racket/private/kw.rkt">expander for optional/keyword functions</a> looks for default expressions vs. default values and uses the undefined value for expressions.</p> + +<p>Three other facts conspired to make the problem with optional methods:</p> + +<ol> + <li>Typed Racket also looks for default expressions vs. default values (search for <strong>immediate-default</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/base-env/annotate-classes.rkt">here</a>). When an optional parameter has a default expression, Typed Racket widens its internal type to accept the <strong>unsafe-undefined</strong> value (search for <strong>-Unsafe-Undefined</strong> <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/types/kw-types.rkt">here (kw)</a> and <a href="https://github.com/racket/typed-racket/blob/325f621716966b95a68af700624bafa21ac66e14/typed-racket-lib/typed-racket/typecheck/tc-lambda-unit.rkt">here (opt)</a>).</li> + <li>The class expander does some pre-processing on optional methods and inadvertantly turned every default value into a default expression.</li> + <li>Shallow TR pushes default expression checks <strong>(if test default-expr supplied-arg)</strong> to the <strong>supplied-arg</strong> instead of wrapping the whole <strong>if</strong> form.</li></ol> + +<p>In the end, Typed Racket saw a default value and inferred an overly-precise type. The type would be correct but for the class expander. As-is, the type was unsound&mdash;but harmless because the false assumption was guarded by an <strong>if</strong> test for <strong>unsafe-undefined</strong>. Running Shallow TR revealed the unsoundness with its eager shape check.</p> + +<p>Again, the resolution was to fix the class expander (<a href="https://github.com/racket/racket/pull/3182">racket/racket #3182</a>). Both Typed Racket and Shallow TR stayed the same. The change removes an unnecessary run-time check from expanded optional methods.</p> + +<h2 id="lessons">Lessons</h2> + +<ol> + <li>Optional and keyword functions are not core forms in Racket. They expand to a combination of simple functions.</li> + <li>Digging into the expansion is sometimes necessary. There are at least three places that do so&mdash;the class expander, TR, and Shallow TR&mdash;and unfortunately they all need to cooperate.</li> + <li>The development of Shallow TR helped find several latent bugs in TR, Racket, and other libraries. Figure 57 of <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-dissertation-2020">my dissertation</a> lists them all.</li></ol> + + Transient Answers Old Questions + http://prl.ccs.neu.edu/blog/2020/10/15/transient-answers-old-questions/?utm_source=typed-racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-10-15-transient-answers-old-questions + Thu, 15 Oct 2020 13:32:12 UT + Ben Greenman + +<p>Several old questions from the Typed Racket mailing list have new and simple answers under a &ldquo;transient&rdquo; Typed Racket.</p> +<!-- more--> + +<hr /> + +<p>For the past few months, I&rsquo;ve been adding a transient semantics to Typed Racket. The project is called Shallow Typed Racket. Details are in the <a href="https://github.com/racket/typed-racket/pull/952">RFC</a> and <a href="https://github.com/racket/typed-racket/pull/948">pull request</a>.</p> + +<p>The short story is that the new Shallow Racket does less to enforce types when typed code interacts with untyped code. Typed code is still type-sound, but that&rsquo;s about it. By contrast, types are much stronger in classic Typed Racket.</p> + +<p>Shallow Racket&rsquo;s weaker types allow more programs to run. While testing whether the new freedom is useful, I reviewed a few years of Typed Racket questions on the <a href="https://groups.google.com/g/racket-users">Racket mailing list</a>. There were a surprising number of questions that went like this:</p> + +<blockquote> + <p><strong>Q.</strong> Hey, I ran a program expecting <em>X</em> to happen, but <em>Y</em> happened instead. Is this a bug?</p> + <p><strong>A.</strong> No, Typed Racket has to do <em>Y</em> because of its strong types.</p></blockquote> + +<p>&hellip; but changing to shallow types gives the <em>X</em> behavior! Here are their stories.</p> + +<p>Going forward, <strong>Deep</strong> refers to normal Typed Racket and <strong>Shallow</strong> refers to Shallow Typed Racket.</p> + +<hr /> + +<h2 id="higher-order-value-as-any">Higher-Order Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ">groups.google.com/g/racket-users/c/cCQ6dRNybDg/m/CKXgX1PyBgAJ</a></p> + +<h4 id="on-20180416-mailoo-wrote">On 2018&ndash;04&ndash;16, <em>mailoo</em> wrote:</h4> + +<blockquote> + <p> I play a little with the &ldquo;Any&rdquo; type (due to &lsquo;dynamic-require&rsquo; which return Any), and I&rsquo;m not able to cast them back in a function.</p> + <p> I (over) simplify my question with this little program :</p></blockquote> + +<pre><code>(: p Any) +(define (p i) (displayln i)) + +; Here I want to get back my function +(define proc (cast p (-&gt; Integer Void))) +(proc 2) </code></pre> + +<blockquote> + <p> but I get this error when I try to execute the function :</p></blockquote> + +<pre><code>; contract violation +; Attempted to use a higher-order value passed as `Any` in untyped code: #&lt;procedure:p&gt; </code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> raises an error because it must enforce the <code>Any</code> type with a contract that rejects all interactions. Things would go badly if an Any-typed function expected a String but got an Integer.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> prints 2 and returns void. No error. Same goes for dynamic-require.</p> + +<hr /> + +<h2 id="parametric-contract-affects-untyped-code">Parametric Contract Affects Untyped Code</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ">groups.google.com/g/racket-users/c/ZbYRQCy93dY/m/kF_Ek0VvAQAJ</a></p> + +<h4 id="on-20191215-john-clements-wrote">On 2019&ndash;12&ndash;15, John Clements wrote:</h4> + +<blockquote> + <p> It looks like my quick attempt at importing index-of into TR is running into a problem. Here’s the program I ran:</p></blockquote> + +<pre><code> #lang typed/racket + + (require/typed racket/list + [index-of (All (T) ((Listof T) T -&gt; (U False Natural)))]) + + (index-of '(n s e w) 'n) ;; returns... #f? </code></pre> + +<blockquote> + <p> In typed/racket/no-check this returns 0, and also in racket (mutatis mutandis).</p> + <p> I thought this might be some kind of parametricity issue, but even when I instantiate index-of at Symbol which should pretty much clear the way for arbitrary equality checking, I still get False.</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces parametricity for <code>All</code> types, and this throws off the equality function that index-of uses internally.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> returns 0.</p> + +<p>ps John, thanks very much for working on <a href="https://adventofcode.com">Advent of Code</a> and mailing the list!</p> + +<hr /> + +<h2 id="unable-to-protect-opaque-value-as-any">Unable to Protect Opaque Value as Any</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ">groups.google.com/g/racket-users/c/jtmVDFCGL28/m/jwl4hsjtBQAJ</a></p> + +<h4 id="on-20191211-marc-kaufmann-wrote">On 2019&ndash;12&ndash;11, Marc Kaufmann wrote:</h4> + +<blockquote> + <p>I have one file called <code>type-test.rkt</code> with the following</p></blockquote> + +<pre><code>#lang typed/racket + +(require (only-in typed/web-server/http response/xexpr response)) + +(provide f2) + +(: f2 (-&gt; (U response Any))) +(define (f2) + (define x '(body (h1 "Try it"))) + (: resp response) + (define resp (response/xexpr x)) + resp)</code></pre> + +<blockquote> + <p>Then I have another <em>untyped</em> file for a servlet:</p></blockquote> + +<pre><code>#lang racket + +(require "type-test.rkt" + web-server/servlet + web-server/servlet-env) + +(define (start req) + (f2)) + +(serve/servlet start + #:servlet-regexp #rx"" + #:launch-browser? #false + #:port 8080)</code></pre> + +<blockquote> + <p>Notice that I am telling [f2] that <code>resp</code> is of type <code>response</code>. Yet, when I run the server with <code>start</code> [&hellip;.] I get the following result:</p> + <p>(f2): Error, see below.</p> + <p>The error is:</p></blockquote> + +<pre><code>f2: broke its own contract + any-wrap/c: Unable to protect opaque value passed as `Any` + value: #&lt;response&gt; + in: the range of + (-&gt; Any)</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the <code>Any</code> type with a contract that rejects all interactions, but needs to know what interactions are possible in order to make a reject-all contract. For many values, Deep can ask questions like procedure? and struct-info to learn enough. But this program sends an opaque response struct across a boundary and Deep does not have the right inspector to learn about the struct fields.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> does nothing to enforce the Any type. This program runs, and in general Shallow never complains about opaque values.</p> + +<hr /> + +<h2 id="type-inference-installs-a-precise-type">Type Inference Installs a Precise Type</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ">groups.google.com/g/racket-users/c/2X5olKMV3C4/m/mJhsp9ZWBgAJ</a></p> + +<h4 id="on-20200214-john-clements-wrote">On 2020&ndash;02&ndash;14, John Clements wrote:</h4> + +<blockquote> + <p>I think I may understand what’s going on here, but a student and I worked on this for quite a while today before I found the problem.</p> + <p>Here’s a program:</p></blockquote> + +<pre><code>#lang typed/racket + +(define-type Store (Mutable-HashTable Integer Value)) +(define-type Value (U Real Boolean String)) + +(define top-store + (cast + (make-hash (list (cons -1 14) (cons 1 #t) (cons 2 #f))) + Store)) + +(hash-set! top-store 5 1234)</code></pre> + +<blockquote> + <p>It fails with this error:</p></blockquote> + +<pre><code>contract violation +expected: (or/c (and/c byte? positive?) #t #f) +given: 1234 +in: the values of +the 3rd conjunct of +(and/c hash? + hash-mutable? + (hash/c exact-integer? + (or/c (and/c byte? positive?) #t #f) + #:immutable #f))</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p>First off, <strong>Deep</strong> runs fine after swapping <code>cast</code> for <code>ann</code>.</p> + +<p>Second, Typed Racket does try to generalize inferred types for mutable data. If the only value in the hash is the byte 14 then Deep also runs.</p> + +<p>The problem is that Typed Racket does not generalize the inferred value type (U Byte Boolean) and that cast is a run-time tool for enforcing types. Casts create contracts to protect mutable data. In this program, there are two contracts: one based on the Store type to protect code that uses the hash, and one based on the inferred type to protect the hash against bad writes. That second contract raises the error message.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> runs successfully. The cast looks for a hash, does not make a contract, and ignores the inferred type going forward.</p> + +<hr /> + +<h2 id="same-arity-functions-in-a-case-lambda">Same-Arity Functions in a Case Lambda</h2> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ">groups.google.com/g/racket-users/c/BDrrgW0axGQ/m/P31NxeGHAAAJ</a></p> + +<h4 id="on-20190705-ryan-kramer-wrote">On 2019&ndash;07&ndash;05, Ryan Kramer wrote:</h4> + +<blockquote> + <p>In the code below, can <code>maybe-car</code> have the given type [&hellip;.]?</p></blockquote> + +<pre><code>#lang typed/racket + +(module untyped racket + (provide maybe-car) + (define (maybe-car x) + (cond + [(pair? x) (car x)] + [else x]))) + +(require/typed + 'untyped + [maybe-car (All (a b) (case-&gt; + (-&gt; (Pairof a b) a) + (-&gt; a a)))])</code></pre> + +<blockquote> + <p>[Current error:]</p></blockquote> + +<pre><code>Type Checker: + Type (All (a b) (case-&gt; (-&gt; (Pairof a b) a) (-&gt; a a))) + could not be converted to a contract: + function type has two cases of arity 1</code></pre> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> tries to enforce the type with a Racket <code>or/c</code> contract, but cannot. The problem is that or/c only has partial support for unions. If or/c ends up with two possible higher-order options at runtime, it halts. In this case, we end up with two function contracts that have the same arity and don&rsquo;t know which to apply to an incoming function.</p> + +<p>Note, the &ldquo;Type Checker&rdquo; error message is much better than what or/c would give on its own.</p> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> simply checks that maybe-car accepts both arities inside the case-&gt; type. The code runs fine. Later, when the function gets applied in typed code, Shallow spot-checks the results.</p> + +<hr /> + +<h3 id="immutable-type-affects-untyped-code">Immutable Type Affects Untyped Code</h3> + +<p>Original message : <a href="https://groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ">groups.google.com/g/racket-users/c/UD20HadJ9Ec/m/Lmuw0U8mBwAJ</a></p> + +<h4 id="on-20200217-bertrand-augereau-wrote">On 2020&ndash;02&ndash;17, Bertrand Augereau wrote:</h4> + +<blockquote> + <p>Hello everybody, I&rsquo;m trying to gradually type my script to make it a proper app (yes I&rsquo;m a static-ish guy) and I have an issue (Racket 7.6 CS).</p></blockquote> + +<pre><code>; racket_mod.rkt: +#lang racket + +(provide (struct-out s)) +(provide list-of-s) +(provide set-list-of-s!) + +(struct s (a)) +(define list-of-s '()) +(define (set-list-of-s! los) + (set! list-of-s los))</code></pre> + +<pre><code>; racket_mod_typed.rkt: +#lang typed/racket + +(provide (struct-out s2)) +(provide list-of-s2) +(provide set-list-of-s2!) + +(struct s2 ([a : Natural])) +(define list-of-s2 '()) +(define (set-list-of-s2! [los : (Listof s2)]) + (set! list-of-s2 los))</code></pre> + +<pre><code>; racket_main.rkt: +#lang racket + +(require "racket_mod.rkt") +(require "racket_mod_typed.rkt") + +(define los (list (s 1) (s 2))) +(set-list-of-s! los) +(displayln list-of-s) + +(define los2 (list (s2 1) (s2 2))) +(set-list-of-s2! los2) +(displayln list-of-s2)</code></pre> + +<blockquote> + <p>list-of-s2 is empty and list-of-s is not, the only difference seems to be the type annotations. Can someone help me ? :)</p></blockquote> + +<h3 id="whats-going-on">What&rsquo;s going on?</h3> + +<p><strong>Deep</strong> enforces the type of <code>list-of-s2</code> with a listof contract, which ends up making a copy of the original (empty) list as it traverses and validates it. The original value does change in typed code, but the main module only has access to the empty copy.</p> + +<p>Here&rsquo;s a step-by-step breakdown:</p> + +<ol> + <li>the typed module creates an empty list-of-s2</li> + <li>the main module imports the list and receives a new copy</li> + <li>the main module calls set-list-of-s2! and the typed module updates the original list-of-s2 variable</li> + <li>the main module reads from its copy &mdash; and it&rsquo;s still empty</li></ol> + +<h3 id="hows-transient">How&rsquo;s transient?</h3> + +<p><strong>Shallow</strong> lets the original list travel to untyped code. There are no contracts in the way.</p> + +<h2 id="discussion">Discussion</h2> + +<p>Wow! It&rsquo;s great to see that Shallow Racket works &ldquo;as expected&rdquo; on these examples. I hope the Shallow option makes types more accessible to more Racket programmers in the future.</p> + +<p>If you have a similar experience with a deep-types error, let me know.</p> + +<p>Keep in mind, though, the freedoms of shallow types allow silent failures. A value can pass by a mis-matched type annotation without Shallow raising an error &mdash; and if that happens, the end result may be really, really confusing. Of course you can always switch back to Deep Typed Racket for debugging.</p> + +<p>Shallow Typed Racket is coming soon. Follow the <a href="https://github.com/racket/typed-racket/pull/948">pull request</a> or watch the Racket release notes for news.</p> + +<h3 id="links">Links</h3> + +<ul> + <li><a href="http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/">Larger example</a> where Shallow misses an error that Deep catches</li> + <li>Michael M. Vitousek <a href="http://hdl.handle.net/2022/23172">invented</a> the Transient semantics and implemented it in <a href="https://github.com/mvitousek/reticulated">Reticulated Python</a>.</li> + <li>My <a href="https://ccs.neu.edu/home/types/publications/publications.html#g-thesis-2020">upcoming dissertation</a> has lots more to say about Shallow Typed Racket.</li></ul> + +<p><em>Thanks to Artem Pelenitsyn for reading and criticizing an early version of this post.</em></p> + + The Typed Racket Optimizer vs. Transient + http://prl.ccs.neu.edu/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/?utm_source=typed-racket&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2020-01-15-the-typed-racket-optimizer-vs-transient + Wed, 15 Jan 2020 12:16:35 UT + Ben Greenman + +<p>What type-directed optimizations does Typed Racket perform and do any require full types?</p> +<!-- more--> + +<blockquote> + <p>This post is based on a short talk. Slides from the talk are here: <a href="http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf">http://ccs.neu.edu/home/types/resources/talks/prl-offsite-2019.pdf</a></p></blockquote> + +<p>Standard Typed Racket guarantees full type soundness and uses higher-order contracts to make sure that interactions between Typed Racket and untyped Racket obey the types. These contracts can be very expensive [<a href="https://doi.org/10.1017/S0956796818000217">JFP 2019</a>]. And so, the standard types are very strong but (possibly) slow.</p> + +<p>Lately, I&rsquo;ve been working on a <a href="https://dl.acm.org/citation.cfm?id=3009849">transient</a> back-end for Typed Racket. Transient Typed Racket provides a weaker guarantee &mdash; only that typed code cannot get &ldquo;stuck&rdquo; &mdash; via simpler run-time checks. Early data shows that these simple checks are often faster than the standard boundary checks [<a href="https://doi.org/10.1145/3236766">ICFP 2018</a>], hence we want both options for Typed Racket programmers: slow/correct and fast/wrong.</p> + +<p>The implementation of Transient needs to re-use some parts of Standard Typed Racket and modify others. Typed Racket comes with three major components:</p> + +<ol> + <li>a static type checker,</li> + <li>a compiler from types to contracts, and</li> + <li>a type-driven optimizer [<a href="https://www2.ccs.neu.edu/racket/pubs/padl12-stff.pdf">PADL 2012</a>, <a href="https://doi.org/10.1145/2384616.2384629">OOPSLA 2012</a>].</li></ol> + +<p>Transient Typed Racket can re-use all of the type checker and parts of the type-to-contract compiler. The question for this post is: can Transient re-use the optimizer?</p> + +<h2 id="q-can-transient-re-use-the-typed-racket-optimizer">Q. Can Transient re-use the Typed Racket optimizer?</h2> + +<p>The answer requires some thought because Standard Typed Racket and Transient Typed Racket preserve different amounts of type information.</p> + +<ul> + <li>In Standard Typed Racket, if an expression <strong>e</strong> has type <strong>T</strong> and reduces to a value <strong>v</strong> (for short, <strong>e : T &mdash;&gt;* v</strong>), then the result <strong>v</strong> definitely matches the full type <strong>T</strong>.</li> + <li>In Transient Typed Racket, if <strong>e : T &mdash;&gt;* v</strong> then the result <strong>v</strong> matches the toplevel &ldquo;shape&rdquo; of <strong>T</strong> but (maybe) nothing more.</li></ul> + +<p>The idea of a &ldquo;shape&rdquo; is that it corresponds to the outermost constructor of a type. A shape check must be decidable, but otherwise finding the best shape for a type is an engineering challenge. On one hand, deeper checks give stronger guarantees. On the other hand, shallower checks are quicker to validate.</p> + +<p>Here are a few shapes according to the current Transient prototype:</p> + +<pre><code> Shape(Natural) = Natural + Shape(Listof String) = Listof Any + Shape(Symbol -&gt; Boolean) = Any -&gt; Any + Shape(Vector Void Void) = Vector Any Any + Shape(U Void (Listof Symbol)) = U Void (Listof Any)</code></pre> + +<p>For the current shapes, can we re-use the Typed Racket optimizer?</p> + +<h2 id="optimization-topics">Optimization Topics</h2> + +<p>Typed Racket implements 15 kinds of type-directed transformation. Below, each gets: a short description, an example, and a verdict of &ldquo;safe&rdquo; or &ldquo;unsafe&rdquo; for Transient.</p> + +<p>To be clear: some optimization topics perform many kinds of transformations, but this post picks only one example transformation for each.</p> + +<hr /> + +<h3 id="topic-1-apply">Topic 1: apply</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/apply.rkt">apply.rkt</a> &ldquo;inlines&rdquo; expressions of the form <code>(apply f (map g xs))</code> to map and fold in one pass over the list (<code>xs</code>). Currently, the pass only triggers when <code>f</code> is <code>+</code> or <code>*</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: xs (Listof Integer)) + + ;; -------------------------------------------------- + ;; Before Optimization + (apply + (map abs xs)) + + ;; -------------------------------------------------- + ;; After Optimization + (let loop ((v 0) + (lst xs)) + (if (null? lst) + v + (loop (+ v (abs (unsafe-car lst))) + (unsafe-cdr lst))))</code></pre> + +<p><strong>Verdict</strong>: safe, but risky.</p> + +<p>Technically, this transformation is unsound for Transient because of how it uses <code>unsafe-car</code>. The expansion of <code>(apply * (map g xs))</code> applies <code>(g (unsafe-car xs))</code> without confirming that the first element of <code>xs</code> matches its expected type. This unsoundness is no problem, though, as long as <em>every</em> Transient-typed function checks the shape of its input. (Typed functions that flow to untyped code already need to check inputs.)</p> + +<hr /> + +<h3 id="topic-2-box">Topic 2: box</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/box.rkt">box.rkt</a> safely applies unsafe box operations to expressions with <code>Box</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b (Boxof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (unbox b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-unbox b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-3-dead-code">Topic 3: dead-code</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/dead-code.rkt">dead-code.rkt</a> uses type information to identify code that cannot run. Once identified, the TR optimizer makes the dead code obvious for the Racket bytecode compiler. The pass deals with <code>if</code> expressions, <code>lambda</code> expressions, and <code>case-lambda</code>; the latter is the most interesting for Transient.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Symbol Symbol) + + ;; -------------------------------------------------- + ;; Before Optimization + (define f + (case-lambda + ((s) s) + ((s i) + (for/list ((_i (in-range i))) s)))) + + ;; -------------------------------------------------- + ;; After Optimization + (define f + (case-lambda + ((s) s) + ((s i) + ; dead code, replace with no-op + (void))))</code></pre> + +<p><strong>Verdict</strong>: unsafe, can change behavior</p> + +<p>The pass infers that some branches of a <code>case-lambda</code> can never run because the type says they do not exist. In Standard Typed Racket, this inference is correct because a run-time contract seals off the &ldquo;untyped&rdquo; branches. In Transient, though, there is no need to add a contract and therefore no guarantee these branches are inaccessible. An application in untyped code can enter the dead branch; if it does, then adding Transient types to part of a program can change its result to <code>(void)</code> and thereby violate the graduality design goal [<a href="http://drops.dagstuhl.de/opus/volltexte/2015/5031/">SNAPL 2015</a>, <a href="https://doi.org/10.1145/3236768">ICFP 2018</a>] &mdash; that is, that adding types should only change behavior by introducing runtime type mismatches.</p> + +<hr /> + +<h3 id="topic-4-extflonum">Topic 4: extflonum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/extflonum.rkt">extflonum.rkt</a> safely applies unsafe extflonum operations to expressions with <code>Extflonum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: e Extflonum) + + ;; -------------------------------------------------- + ;; Before Optimization + (extflabs e) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-extflabs e)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-5-fixnum">Topic 5: fixnum</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/fixnum.rkt">fixnum.rkt</a> safely applies unsafe fixnum operations to expressions with <code>Fixnum</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f Fixnum) + + ;; -------------------------------------------------- + ;; Before Optimization + (exact-&gt;inexact f) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-fx-&gt;fl f)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-6-float-complex">Topic 6: float-complex</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float-complex.rkt">float-complex.rkt</a> unboxes complex numbers (into one real-part variable and one imaginary-part variable) and rewrites operations to handle the unboxed numbers.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: f (-&gt; Float-Complex Float-Complex Float-Complex)) + + ;; -------------------------------------------------- + ;; Before Optimization + (define (f n0 n1) + (+ n0 n1)) + + ;; -------------------------------------------------- + ;; After Optimization + (define (f n0 n1) + (let* ((unboxed-real-0 (unsafe-flreal-part n0)) + (unboxed-imag-0 (unsafe-flimag-part n0)) + (unboxed-real-1 (unsafe-flreal-part n1)) + (unboxed-imag-1 (unsafe-flimag-part n1)) + (unboxed-real-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-real-0) + unboxed-real-1)) + (unboxed-imag-2 (unsafe-fl+ (real-&gt;double-flonum unboxed-imag-0) + unboxed-imag-1))) + (unsafe-make-flrectangular unboxed-real-2 unboxed-imag-2)))</code></pre> + +<p><strong>Verdict</strong>: safe, with caution</p> + +<p>The body of a Transient-typed function (that can flow to untyped code) must first check that its inputs have the correct shape. Currently, the <strong>float-complex</strong> pass creates functions that apply <code>unsafe-flreal-part</code> before anything else; to be safe, the pass needs to make sure that Transient checks come first.</p> + +<hr /> + +<h3 id="topic-7-float">Topic 7: float</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/float.rkt">float.rkt</a> safely applies unsafe flonum operations to expressions with <code>Flonum</code> type and also transforms some <code>random</code> calls to use <code>unsafe-flrandom</code>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; -------------------------------------------------- + ;; Before Optimization + (random) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-flrandom (current-pseudo-random-generator))</code></pre> + +<p><strong>Verdict</strong>: safe, but a close call</p> + +<p>Accessing a parameter, as in <code>(current-pseudo-random-generator)</code>, is an elimination form that may require a shape check. This particular parameter, however, is protected by a contract that enforces the precondition of <code>unsafe-flrandom</code>.</p> + +<hr /> + +<h3 id="topic-8-list">Topic 8: list</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/list.rkt">list.rkt</a> safely applies unsafe list operations to list expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: lst (List Symbol Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (list-ref lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-list-ref lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The shape check for a <code>(Listof T)</code> must check for proper lists (via <code>list?</code>); note that the cost of this check depends on the size of incoming values. The shape check for a <code>(List T ...)</code> type must validate the length of incoming values.</p> + +<hr /> + +<h3 id="topic-9-number">Topic 9: number</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/number.rkt">number.rkt</a> performs simple transformations on <code>Real</code>-valued expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: r Real) + + ;; -------------------------------------------------- + ;; Before Optimization + (+ r) + + ;; -------------------------------------------------- + ;; After Optimization + r</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-10-pair">Topic 10: pair</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/pair.rkt">pair.rkt</a> safely applies pair-access operations to (possibly-nested) pairs.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: p (Pairof (Pairof Symbol Void) String)) + + ;; -------------------------------------------------- + ;; Before Optimization + (cdar p) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-cdr (unsafe-car p))</code></pre> + +<p><strong>Verdict</strong>: unsafe</p> + +<p>Transient guarantees the first level of a type, but nothing more. Concretely, <code>Shape(Pairof (Pairof Symbol Void) String) = Pairof Any Any</code> and so the <code>unsafe-cdr</code> above is not safe.</p> + +<hr /> + +<h3 id="topic-11-sequence">Topic 11: sequence</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/sequence.rkt">sequence.rkt</a> safely applies unsafe sequence operations to expressions with <code>(Sequenceof T)</code> type.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: s String) + + ;; -------------------------------------------------- + ;; Before Optimization + (for ((c s)) + (void)) + + ;; -------------------------------------------------- + ;; After Optimization (simplified) + (for ((c (in-string s))) + (void))</code></pre> + +<p><strong>Verdict</strong>: safe, with strong enough shape checks (see <strong>list</strong> and <strong>vector</strong>)</p> + +<hr /> + +<h3 id="topic-12-string">Topic 12: string</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/string.rkt">string.rkt</a> safely applies unsafe string operations to expressions with <code>String</code> type. (Note that <code>unsafe-string-ref</code> is only safe when the result is sure to be a Latin&ndash;1 character.)</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: b Bytes) + + ;; -------------------------------------------------- + ;; Before Optimization + (bytes-length b) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-bytes-length b)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-13-struct">Topic 13: struct</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/struct.rkt">struct.rkt</a> safely applies unsafe struct operations to struct expressions, using Typed Racket&rsquo;s <a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/types/struct-table.rkt">internal registry of struct info</a>.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (struct open-interval ([lo : Real] [hi : Real])) + (: ivl open-interval) + + ;; -------------------------------------------------- + ;; Before Optimization + (open-interval-lo ivl) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-struct-ref ivl 0)</code></pre> + +<p><strong>Verdict</strong>: safe</p> + +<hr /> + +<h3 id="topic-14-unboxed-let">Topic 14: unboxed-let</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/unboxed-let.rkt">unboxed-let.rkt</a> cooperates with the <code>float-complex</code> pass by transforming the binding-site of some complex numbers. This pass may change a <code>let</code>-expression into a <code>let-values</code> that expects a real-part and imag-part, and may change a function to expect twice as many arguments &mdash; provided the optimizer can find <em>all</em> calls to the function.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: k Float-Complex) + + ;; -------------------------------------------------- + ;; Before Optimization + (let ((f (lambda ((n : Float-Complex)) (+ n n)))) + (f k)) + + ;; -------------------------------------------------- + ;; After Optimization + (let ((f (lambda (real-part-n imag-part-n) ....))) + (f (unsafe-flreal-part k) (unsafe-flimag-part k)))</code></pre> + +<p><strong>Verdict</strong>: safe, thanks to the (conservative) escape analysis</p> + +<hr /> + +<h3 id="topic-15-vector">Topic 15: vector</h3> + +<p><a href="https://github.com/racket/typed-racket/blob/master/typed-racket-lib/typed-racket/optimizer/vector.rkt">vector.rkt</a> safely applies vector operations to vector expressions.</p> + +<h4 id="example">Example</h4> + +<pre><code> ;; Type Assumptions + (: v (Vector (Listof Symbol) String)) + (: lst (Listof Symbol)) + + ;; -------------------------------------------------- + ;; Before Optimization + (vector-set! v lst 0) + + ;; -------------------------------------------------- + ;; After Optimization + (unsafe-vector-set! v lst 0)</code></pre> + +<p><strong>Verdict</strong>: safe, with strong-enough shape checks</p> + +<p>The check for <code>(Vector T ...)</code> must check the length of incoming values.</p> + +<hr /> + +<h2 id="summary">Summary</h2> + +<p>The Typed Racket optimizer implements 15 kinds of transformations. Two are definitely unsafe for Transient as-is (<strong>dead-code</strong>, <strong>pair</strong>). One must take care when rewriting a Transient function (<strong>float-complex</strong>). One may limit our ability to reduce the number of run-time checks in a program (<strong>apply</strong>). Two others require transient checks whose cost depends on the size of the input values (<strong>list</strong>, <strong>sequence</strong>).</p> + +<p>There may be other issues that I missed while reading the optimizer code. If so, I&rsquo;ll try to remember to update this post.</p> \ No newline at end of file diff --git a/blog/feeds/types.atom.xml b/blog/feeds/types.atom.xml new file mode 100644 index 00000000..6486a184 --- /dev/null +++ b/blog/feeds/types.atom.xml @@ -0,0 +1,144 @@ + + + PRL Blog: Posts tagged 'types' + + + urn:http-prl-ccs-neu-edu:-blog-tags-types-html + 2017-10-22T11:59:06Z + + Monotonicity Types: Towards A Type System for Eventual Consistency + + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + 2017-10-22T11:59:06Z + 2017-10-22T11:59:06Z + + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> \ No newline at end of file diff --git a/blog/feeds/types.rss.xml b/blog/feeds/types.rss.xml new file mode 100644 index 00000000..113df3c8 --- /dev/null +++ b/blog/feeds/types.rss.xml @@ -0,0 +1,144 @@ + + + + PRL Blog: Posts tagged 'types' + PRL Blog: Posts tagged 'types' + http://prl.ccs.neu.edu/blog/tags/types.html + Sun, 22 Oct 2017 11:59:06 UT + Sun, 22 Oct 2017 11:59:06 UT + 1800 + + Monotonicity Types: Towards A Type System for Eventual Consistency + http://prl.ccs.neu.edu/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/?utm_source=types&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-10-22-monotonicity-types-towards-a-type-system-for-eventual-consistency + Sun, 22 Oct 2017 11:59:06 UT + Kevin Clancy + +<p>A few weeks back, we published a draft of an article entitled <a href="https://infoscience.epfl.ch/record/231867"><em>Monotonicity Types</em></a>. In it, we describe a type system which we hope can aid the design of distributed systems by tracking monotonicity with types.</p> +<!-- more--> + +<p>But first, what, precisely, do we mean by <em>monotonicity</em>? Here&rsquo;s a short definition:</p> + +<p>A partially ordered set is a set \(P\) endowed with a relation \(\leq\) such that for all \(p, q, r \in P\) we have:</p> + +<ol> + <li>\(p \leq p\) (reflexivity)</li> + <li>\(p \leq q\) and \(q \leq r\) implies \(p \leq r\) (transitivity)</li> + <li>\(p \leq q\) and \(q \leq p\) implies \(p = q\) (anti-symmetry)</li></ol> + +<p>If \(P\) and \(Q\) are partially ordered sets, we say that a function \(f : P \to Q\) between them is <em>monotone</em> if for all \(p_1, p_2 \in P\) with \(p_1 \leq p_2\), we have \(f(p_1) \leq f(p_2)\).</p> + +<p>So, said another way, increasing the input to a monotone function causes an increase to its output.</p> + +<p>Particularly in the context of concurrent and distributed programming, monotonicity has arisen time and time again as an important property. Designers of languages for coordination-free distributed programming such as Lasp [<a href="#ref1">Meiklejohn et al. (2015)</a>] and BloomL [<a href="#ref1">Conway et al. (2012)</a>], as well as designers of data types and abstractions for eventual consistency or determinism such as CRDTs [<a href="#ref3">Shapiro et al. (2011)</a>] and LVars [<a href="#ref4">Kuper et al. (2013)</a>] have noticed that monotonic evolution of program state over time is a necessary property in their designs. Lasp and BloomL in particular require the use of monotone functions as primitives of program composition.</p> + +<p>Thus if a user would like to make use of such a language for concurrent and distributed programming, they&rsquo;re required to write monotonic program functions, which can actually be quite tricky, in order to get the consistency or determinism guarantees that the given language/abstraction was designed to provide.</p> + +<p>To get a better idea of how monotonicity might be important in the context of data replicated over a distributed system, let&rsquo;s look at an example. Suppose we need a function to determine whether a replicated counter&rsquo;s current value is odd or even, and further suppose that this counter can only be incremented. To accomplish this, we might apply the following function to the counter&rsquo;s value:</p> + +<pre><code>fun IsOdd(x : Nat) = x % 2 == 1</code></pre> + +<p>However, the counter replica from which the argument x is obtained may not currently have an up-to-date count of the total number of increments performed in the entire system. We can&rsquo;t rule out the possibility that exactly one remote increment has been performed, in which case IsOdd produces the wrong answer. With this in mind, the value returned by IsOdd does not seem to tell us anything useful. In contrast, consider an application of the following function to the same replicated counter.</p> + +<pre><code>fun MoreThanTen(x : Nat) = x &gt; 10</code></pre> + +<p>The boolean values \(true\) and \(false\) form one of the simplest partially ordered sets of all. We consider \(false \leq false\), \(false \leq true \), and \( true \leq true \). Under this ordering, the MoreThanTen function is monotone: an increase in x can cause the value of \(x &gt; 10\) to flip from false to true, but not vice versa. When we observe that the local counter replica P&rsquo;s value is greater than 10, we don&rsquo;t know that the same observation would be drawn from remote replicas. Nonetheless, we assume that all replicas in the system will eventually become aware of all increments that P is currently aware of, at which point their values will be greater than P&rsquo;s current value. This is where MoreThanTen&rsquo;s monotonicity becomes useful. At the point when all replicas have received P&rsquo;s current information, every replica in the system will agree that MoreThanTen applied to the counter&rsquo;s value returns true.</p> + +<p>We believe that a type system for proving functions monotone could push the development of coordination-free distributed and concurrent applications outside of the realm of distributed systems experts, by enabling customization and extension of such systems by non-experts.</p> + +<p>Towards this aim, we have been designing a typed lambda calculus in which types track monotonicity. Our approach allows the programmer to write a special kind of function definition, called an <em>sfun</em>, the body of which is type checked using a richer type system, one which reasons about function composition rather than application. Such a function can be proven monotone by utilizing, among other principles, the fact that the composition of two monotone functions is itself monotone. Monotonicity is a relational property; that is, its a property involving multiple applications of the same function. Such properties are blind spot for traditional type systems, so our design requires some unusual and interesting features.</p> + +<p>Reasoning about pointwise orderings on function spaces seems a bit heavy-weight and hasn’t been necessary for any of my use cases. An sfun is therefore first order; that is, both its return type and all of its argument types must be data types rather than function types. We would like to be able to prove that a multi-argument function is monotone <em>separately</em> in each of its arguments; that is, for \(i \in 1..n\), if \(p_i \leq p_i'\) then \(f(p_1, \ldots, p_i, \ldots, p_n) \leq f(p_1, \ldots p_i', \ldots p_n)\).</p> + +<p>The monotonicity of an sfun is typically derived from the monotonicity of the primitives used to implement it, which are also sfuns. Here are some example sfun primitives, addition and subtraction on integers:</p> + +<p>1.) plus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \uparrow y] \)</p> + +<p>2.) minus : \( (x : Int, y : Int) \Rightarrow Int[\uparrow x, \downarrow y] \)</p> + +<p>An <em>sfun type</em>, written with \(\Rightarrow\) rather than \(\rightarrow\), names its formal arguments and also <em>qualifies</em> each one. A qualifier is an argument-specific constraint on the behavior of the function. In the above types, the qualifier \(\uparrow\) is associated with arguments that are separately monotone and \(\downarrow\) is associated with arguments that are separately antitone. The second argument of a binary function \(f\) is separately antitone if \(p_2 \leq p_2'\) implies \(f(p_1, p_2) \geq f(p_1, p_2')\).</p> + +<p>Terms outside of sfun abstractions are typed using a <em>global</em> typing relation, which, aside from an sfun abstraction typing rule, is not different from the typing relations we are familiar with. A global typing judgment has the following form.</p> + +<p>\( \Gamma \vdash t : T \)</p> + +<p>A typing judgment of the lifted type system, used to type check the body of an sfun, has the following form:</p> + +<p>\( \Gamma;\Omega;\Phi \vdash t : T \)</p> + +<p>Here the <em>global type environment</em> \( \Gamma \) contains all of the variables bound outside of the sfun, the <em>ambient type environment</em> \( \Omega \) contains the list of the sfun’s formal arguments, and the <em>lifted type environment</em> \( \Phi \) contains those variables in \( t \)’s context which are bound inside the sfun. Before getting into the significance of lifted typing judgments, let&rsquo;s look at a specific application of the global typing rule for sfun abstractions, which uses a single lifted premise.</p> + +<p>$$\frac{\Gamma;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x]} {\Gamma \vdash \tilde{\lambda} x : Int. plus(x,x) : ( x : Int ) \Rightarrow Int[\uparrow~x]}$$</p> + +<p>Here we type a single-argument sfun abstraction \(\tilde{\lambda} x:Int. plus(x,x)\). As you might have guessed, \(\tilde{\lambda}\) is used rather that \(\lambda\) to distinguish this as an sfun abstraction rather than a standard one. Examine the ambient and lifted type environments used in the premise. Perhaps surprisingly, the abstraction&rsquo;s bound variable \(x\) is entered into both environments. When variables occur in types, they are considered references to formal arguments rather than actual arguments; that is, an occurrence of \(x\) in a type (for example \(Int[\uparrow x]\)) does not refer to some integer, but instead a &ldquo;slot&rdquo; named \(x\) which expects to receive some integer from an external source. Inside the scope of the sfun abstraction, we would like the ability to refer to the abstraction&rsquo;s formal argument \(x\), and therefore we add \(x : Int\) to the ambient environment. We would also like to include occurrences of \(x\) as terms in the body of the abstraction; for these, we add the entry \(x : Int[=~x]\) into the lifted type environment, to be used as a placeholder for the actual argument supplied to the formal argument \(x\). Because references to formal arguments occur only in types, and references to actual arguments occur only in terms, we can add entries with the same name to both the ambient and lifted environments without creating any ambiguity. In particular, this means that the occurrence of \(x\) in Int[\(\uparrow x\)] refers to the entry for \(x\) in the ambient type environment rather than the one in the lifted type environment.</p> + +<p>The premise of the above rule application includes the strange looking types \(Int[=~x]\) and \(Int[\uparrow~x]\). Normally, we would expect occurrences of x, which serve as placeholders for the actual argument of the the function, to have type \(Int\), and we would expect our abstraction&rsquo;s body \(plus(x,x)\) to have type \(Int\) as well. This traditional approach to typing a function abstraction characterizes the operational behavior of a single function <em>after</em> it has been applied. Unfortunately, this isn&rsquo;t adequate for reasoning about properties such as monotonicity, which involve multiple calls to the same function. My approach instead takes the perspective of inside of a function, <em>before</em> it has been applied. Lifted typing then characterizes the structure of a function as the composition of its constituent parts. In the above example, an occurrence of the variable \(x\) in the term \(plus(x,x)\) has type \(Int[=~x]\), meaning that it is a function which takes the value provided to \(x\) (the enclosing sfun&rsquo;s formal argument) as an input, and produces that value unchanged as a result. We ultimately care about the input/output relation of this function, and so the concrete values which inhabit this type are set-of-pairs function representations, called <em>ambient maps</em>. The type \(Int[=~x]\) happens to be a singleton type, containing the set of pairs \(\{ (0,0), (1,1), (-1,-1), (2,2), (-2-2), \ldots \}\).</p> + +<p>The sfun application \(plus(x,x)\) is viewed as a function composition, where the outputs of the functions represented by the two occurrences of \(x\) are forwarded into the left and right arguments of the sfun \(plus\). The domain of this composite function matches the domain \(x:Int\) of the enclosing sfun, which it inherits from the two occurrences of \(x\). Since \(plus\) returns an \(Int\), so does the composite function \(plus(x,x)\). The premise of the above typing rule application tells us that \(plus(x,x)\) has type \(Int[\uparrow~x]\), but this premise must be derived. We previously hinted that such a derivation may utilize the fact that the composition of two monotone functions is itself monotone, and indeed that is one aspect of the premise&rsquo;s derivation, but a full treatment is outside the scope of this post.</p> + +<p>Since lifted typing is all about function composition, one might wonder how we treat occurrences of \( \Gamma \)&rsquo;s variables within the body of an sfun. Such a variable might have the type \( Int \), representing a data value rather than a function. In fact, a piece of data can be viewed as a degenerate, constant-valued function, which produces the same result regardless of which actual arguments any particular sfun is applied to. Subtyping rules enable the flexible use of terminal variables within the body of an sfun, permitting a variable of type \( Int \), for example, to occur in a context where terms of type \( Int[ \uparrow x ] \) are expected. A constant function \(f\), after all, is monotone: \( v_1 \leq v_2 \) implies \( f(v_1) = c \leq c = f(v_2) \).</p> + +<p>We&rsquo;re not building lifted typing derivations just for fun. Typically, a type system comes with a soundness theorem stating that whenever a typing judgment of the form \( \Gamma \vdash t : T \) is derivable, the execution of the term \(t\) (a program) under some well-defined model of computation (typically defined along with the type system) satisfies some desirable property. In our system, a terminal typing derivation \( \Gamma \vdash t : T \) implies that when the free variables of t are substituted with appropriately-typed values, the execution of the term \( t \) is guaranteed to terminate, producing a value of type \(T\) as its result. This is not a terribly unusual soundness guarantee. However, to provide semantics for lifted typing judgments, we introduced a new reduction relation (or &ldquo;computation model&rdquo;) which can be viewed in one of two ways:</p> + +<ol> + <li>The simultaneous reduction of an sfun, under terminal reduction, when applied to all sets of arguments in its domain.</li> + <li>The composition of an sfun&rsquo;s components, before the sfun is ever applied.</li></ol> + +<p>Point 1 is essentially the motivation for having lifted typing and lifted reduction in the first place. We want to know how the sfun behaves under terminal reduction, across multiple applications&mdash;specifically two applications in the case of monotonicity. If the lifted reduction of an sfun&rsquo;s body faithfully simulates the terminal reduction of all possible applications simultaneously, then the body of a well-typed sfun should normalize to an ambient map that is extensionally equivalent to the sfun&rsquo;s applicative behavior under terminal reduction. Therefore, if our soundness theorem guarantees that the derivability of \( \cdot;x:Int;x:Int[=~x] \vdash plus(x,x) : Int[\uparrow~x] \) implies that \( plus(\{ (0,0), (1,1), \ldots \},\{ (0,0), (1,1), \ldots \} ) \) normalizes under lifted reduction to a monotone ambient map, we then know that the sfun \( \tilde{\lambda} x : Int. plus(x,x) \) behaves monotonically under terminal reduction. It&rsquo;s important to note that our model never requires us to actually perform lifted reduction; lifted reduction matters because not because we actual want to perform it, but instead because lifted typing derivations guarantee the existence of certain lifted reduction sequences which have implications for terminal reduction.</p> + +<p>Point 2 inspires our lifted type system. If an sfun is composed of monotone functions, we can use facts about preservation of monotonicity across function composition to prove the sfun itself monotone. The difference between terminal reduction and lifted reduction is demonstrated by the two mathematical expressions \( f(g(v)) \) and \( (f \circ g) (v) \). The expression \( f(g(v)) \) presents function composition as viewed by a standard type systems: to apply the composition of \(f\) and \(g\) to a value \(v\), we first apply \(g\) to \(v\), and then apply \(f\) to the result. This isn&rsquo;t wrong, but if \( f \) and \( g \) are both monotone, the monotonicity of the composite function as a whole becomes self-evident if we first perform the &ldquo;lifted reduction step&rdquo; \( f(g(v)) \to (f \circ g) (v) \).</p> + +<p>We&rsquo;ll leave you with an aspirational example, which demonstrates the need for a type system, rather than a more monolithic form of analysis, for proving functions monotone. Recall our replicated counter example from the introduction. It isn&rsquo;t sufficient to store this counter as an integer. The problem is that replicas cannot synchronize properly without knowing which how many increments were performed at each replica. Suppose that replicas X and Y each start with a count of zero. The following actions are then performed:</p> + +<ol> + <li>X increments, resulting in a count of 1</li> + <li>X sends a synchronization message to Y, containing X&rsquo;s count 1</li> + <li>X receives a synchronization message from Y containing a count of 1</li></ol> + +<p>At stage 3, X does not know if the received message was sent from Y before or after Y received the synchronization message from stage 2. Replica X therefore does not know whether to set its count to 1 or 2. To avoid this problem, a replicated counter is commonly represented as a map, which maps each replica identifier (a natural number) to the number of increments that replica has performed (also a natural number). It is assumed that any replica id not contained in the map&rsquo;s finite representation maps to 0. Such counters are called GCounters, and described in detail by [<a href="#ref3">Shapiro et al. (2011)</a>].</p> + +<p>GCounters are partially ordered componentwise. We write \( v[a] \) for the natural number to which the GCounter \(v\) maps the replica identifier \(a\), and we write \( \leq \) for the standard ordering on natural numbers. The partial order \( \leq' \) on GCounters is then defined such that \( v \leq' w \) whenever for all replica identifiers \(a\) we have \( v[a] \leq w[a] \).</p> + +<p>[<a href="#ref1">Meiklejohn et al. (2015)</a>] motivates combinators for replicated data types such as the GCounter, but requires that such combinators are monotone separately in each argument. Below is psuedocode for a monotone GCounter addition combinator, annotated with monotonicity types. NatMap is used as the type of maps from natural numbers to natural numbers. Several primitives are defined for working with NatMap. getAt retrieves the kth element of a NatMap m. joinAt returns a new NatMap which is equal to the argument m, except that it maps k to the maximum of m[k] and n. span returns the greatest key mapping to a non-zero value. emptyMap is a NatMap which maps every natural number to 0. + and &gt; are standard arithmetic operators for working with natural numbers.</p> + +<pre><code>getAt :: (m : NatMap, k : Nat) ⇒ Nat[↑ m, ? k] +joinAt :: (m : NatMap, k : Nat, n : Nat) ⇒ NatMap[↑ m, ? k, ↑ n] +span :: (m:NatMap) ⇒ Nat[↑ m] +max :: (a : Nat, b : Nat) ⇒ Nat[↑ a, ↑ b] +emptyMap :: NatMap ++ :: (x:Nat, y:Nat) ⇒ Nat[↑ x, ↑ y] +&gt; :: (x:Nat, y:Nat) ⇒ Bool[↑ x, ↓ y] + +type GCounter = { map : NatMap } + +sfun sumCounters(x : GCounter, y : GCounter) + : GCounter[↑ x, ↑ y] = + let xMap : NatMap[↑ x, ↑ y] = x.map + let yMap : NatMap[↑ x, ↑ y] = y.map + let maxSpan : Nat[↑ x, ↑ y] = max (span xMap) (span yMap) + fun sumCell(k : Nat, acc : NatMap[↑ x, ↑ y]) + : NatMap[↑ x, ↑ y] = + let cond : Bool[↑ x, ↓ y] = k &gt; maxSpan + if cond then + acc + else + let acc' = joinAt acc k ((getAt xMap k) + (getAt yMap k)) + sumCell (k+1) acc' + let initMap : IntArray[↑ x, ↑ y] = emptyMap + GCounter { map = sumCell 0 initMap }</code></pre> + +<p>While our system can handle much of this example, it can&rsquo;t handle everything yet, for several reasons. First, it involves an if condition which depends on the arguments of the enclosing sfun. To handle this, we would need to incorporate the notion of domain restriction into lifted reduction. Second, it involves recursion. This is problematic for us, because our system utilizes the fact that all well-typed programs terminate. We could partially address this by adding terminating fixpoint combinators, which allow recursion given some well-founded termination metric, as in [<a href="#ref5">Vazou et al. (2014)</a>]. However, that would not be adequate for this particular function. Since it could require arbitrarily many levels of recursion depending on which values are supplied as arguments, lifted reduction, which simulates an application to all arguments simultaneously, would diverge.</p> + +<p>So there&rsquo;s still much to do! If you&rsquo;re interested in more details behind the type system, have a look at Kevin&rsquo;s blog article, <a href="https://kevinclancy.github.io/2017/11/09/monotonicity-through-types.html">Monotonicity Through Types</a>, or have a look at the full <a href="https://infoscience.epfl.ch/record/231867">Monotonicity Types</a> preprint for more.</p> + +<h3 id="references">References</h3> + +<p><span id="ref1">C. Meiklejohn and P. Van Roy. <em>Lasp: A language for distributed, coordination-free programming.</em> In Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, PPDP ’15, pages 184–195, New York, NY, USA, 2015. ACM.</span></p> + +<p><span id="ref2">N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. <em>Logic and lattices for distributed programming</em>. In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, pages 1:1–1:14, New York, NY, USA, 2012. ACM.</span></p> + +<p><span id="ref3">M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. <em>Conflict-Free replicated data types</em>. In Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, pages 386–400. Springer, Berlin, Heidelberg, Oct. 2011.</span></p> + +<p><span class="ref4">L. Kuper and R. R. Newton. <em>LVars: Lattice-based data structures for deterministic parallelism</em>. In Proceedings of the 2nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM.</span></p> + +<p><span class="ref5">N. Vazou, E. L. Seidel, R. Jhala, D. Vytiniotis, and S. Peyton-Jones. <em>Refinement types for Haskell</em>. SIGPLAN Not. 49, 9 (August 2014), 269&ndash;282.</span></p> \ No newline at end of file diff --git a/blog/feeds/visr.atom.xml b/blog/feeds/visr.atom.xml new file mode 100644 index 00000000..69552959 --- /dev/null +++ b/blog/feeds/visr.atom.xml @@ -0,0 +1,252 @@ + + + PRL Blog: Posts tagged 'visr' + + + urn:http-prl-ccs-neu-edu:-blog-tags-visr-html + 2022-01-06T17:56:08Z + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + 2022-01-06T17:56:08Z + 2022-01-06T17:56:08Z + + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> \ No newline at end of file diff --git a/blog/feeds/visr.rss.xml b/blog/feeds/visr.rss.xml new file mode 100644 index 00000000..e5e1d22b --- /dev/null +++ b/blog/feeds/visr.rss.xml @@ -0,0 +1,252 @@ + + + + PRL Blog: Posts tagged 'visr' + PRL Blog: Posts tagged 'visr' + http://prl.ccs.neu.edu/blog/tags/visr.html + Thu, 06 Jan 2022 17:56:08 UT + Thu, 06 Jan 2022 17:56:08 UT + 1800 + + Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript) + http://prl.ccs.neu.edu/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/?utm_source=visr&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2022-01-06-introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript + Thu, 06 Jan 2022 17:56:08 UT + Leif Andersen + +<p> + <style>.caption { display: none; }</style></p> + +<p>Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:</p> + +<div class="figure"><img src="/img/intro-visr/visr-and-text.png" alt="Graphical network embedded in text" /> + <p class="caption">Graphical network embedded in text</p></div> + +<p>Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:</p> + +<ul> + <li>version control, such as git, works with interactive-syntax;</li> + <li>programs using interactive-syntax can be written and edited with your favorite text editor or IDE;</li> + <li>cut/copy/paste works with interactive-syntax using your operating system&rsquo;s native clipboard;</li> + <li>code analysis tools, like diff and refactor, still work with interactive-syntax; and</li> + <li>you can use interactive-syntax in any language or environment that supports language-oriented programming.</li></ul> + +<p>To learn more about interactive-syntax, watch <a href="https://www.youtube.com/watch?v=8htgAxJuK5c">this video</a> or read <a href="https://dl.acm.org/doi/10.1145/3428290">the accompanying paper</a>.</p> + +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8htgAxJuK5c" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="allowfullscreen"></iframe> + +<p><a href="https://visr.pl">VISr (Visual and Interactive-Syntax realized) for ClojureScript</a> is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports <a href="https://www.npmjs.com/">NPM packages</a>.</p> + +<p>This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.</p> +<!-- more--> + +<h1 id="getting-started-with-visr">Getting started with VISr</h1> + +<p>Start by going to <a href="https://visr.pl">visr.pl</a>, which is a web-based IDE that directly supports VISrs. Once in the IDE, press <code>Insert VISr</code> to place a VISr at the current cursor position. This VISr contains two buttons:</p> + +<ul> + <li>clicking the first displays the VISr&rsquo;s visual representation, and</li> + <li>clicking the second shows its textual representation.</li></ul> + +<div class="figure"><img src="/img/intro-visr/visr.png" alt="VISr" /> + <p class="caption">VISr</p></div> + +<p>Opening the code shows that the new VISr is an instance of <code>visr.core/empty-visr</code>, a default VISr provided by the IDE. This VISr expects a map with the key <code>:message</code> to display in the visual view. Changing the value associated with <code>:message</code> changes what is displayed, in this case &ldquo;Endless Possibility&rdquo;:</p> + +<div class="figure"><img src="/img/intro-visr/visr-open.png" alt="Open Visr" /> + <p class="caption">Open Visr</p></div> + +<p>Remember that, although this VISr is displayed graphically, it still exists as human-readable text. One way to see this text is by copying and pasting the VISr. A copy of the same VISr will appear when it is placed back into the IDE. However, pasting it into other text editors that do not natively support VISrs yields the following human readable, and editable, text:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="o">^</span><span class="p">{</span><span class="ss">:editor</span> <span class="nv">visr.core/empty-visr</span><span class="p">}(</span><span class="nf">visr.core/empty-visr+elaborate</span> + <span class="p">{</span><span class="ss">:message</span> <span class="s">"Endless Possibility"</span><span class="p">})</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This operation works in reverse too. Writing out similar text and pasting it into <a href="https://visr.pl">visr.pl</a> yields its visual representation.</p> + +<h1 id="making-a-new-visr">Making a new VISr</h1> + +<p>The <code>defvisr</code> form creates a VISr type. This form expects two methods:</p> + +<ol> + <li>a <code>render</code> method that provides visualization and interaction when code is edited, and</li> + <li>an <code>elaborate</code>/<code>elaborate-fn</code> method that gives the VISr compile-time and run-time semantics.</li></ol> + +<p>The following is the signature for a simple VISr type:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span> +<span class="normal">3</span> +<span class="normal">4</span> +<span class="normal">5</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span><span class="p">)</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-render"</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This example uses <code>elaborate-fn</code>, a simplified version of <code>elaborate</code> that gives the <code>VISr</code> the same semantics as a function application. It also allows the <code>defvisr</code> form to work in the same file as the VISr itself.</p> + +<div class="figure"><img src="/img/intro-visr/sig.png" alt="Example of elaborate-fn semantics" /> + <p class="caption">Example of elaborate-fn semantics</p></div> + +<h1 id="the-render-method-for-edit-time-semantics">The Render Method for Edit-Time Semantics</h1> + +<p>The <code>render</code> method is given the VISr state <a href="https://clojure.org/reference/atoms">as an atom</a>; updating this atom also updates the code to reflect the new state. The return value for <code>render</code> must be a <a href="https://reagent-project.github.io/">Reagent form</a> that is the visual view for the VISr. A render method for a counter VISr might look as follows:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span> +<span class="normal">2</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">[</span><span class="ss">:button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nv">this</span> <span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">this</span><span class="p">])</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>And in action:</p> + +<div class="figure"><img src="/img/intro-visr/simpl-count.png" alt="Simple Count Example" /> + <p class="caption">Simple Count Example</p></div> + +<p>This VISr doesn&rsquo;t match the theme of the page; it also requires the state to be a single number. Using <a href="https://react-bootstrap.github.io/">React Bootstrap</a> and Reagent cursors fixes both of these issues:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">example.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="s">"TODO-elaborate"</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="elaboration-and-run-time-semantics">Elaboration and Run-Time Semantics</h1> + +<p>The elaborate method takes the VISr state, and is expected to provide a compile-time or run-time semantics. In the simplified case of <code>elaborate-fn</code>, the VISr semantics takes the form of a function application:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>This <code>elaborate</code> method expects a dictionary with the key <code>:count</code> and returns the value associated with that key. It makes use of <a href="https://clojure.org/guides/destructuring">ClojureScript&rsquo;s Destructuring</a> for brevity. The following code is equivalent:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal">1</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> <span class="p">(</span><span class="nb">get </span><span class="nv">this</span> <span class="ss">:count</span><span class="p">))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<h1 id="putting-it-all-together">Putting it all together</h1> + +<p>The final result is:</p> + +<div class="brush: clojurescript"> + <table class="sourcetable"> + <tbody> + <tr> + <td class="linenos"> + <div class="linenodiv"> + <pre><span class="normal"> 1</span> +<span class="normal"> 2</span> +<span class="normal"> 3</span> +<span class="normal"> 4</span> +<span class="normal"> 5</span> +<span class="normal"> 6</span> +<span class="normal"> 7</span> +<span class="normal"> 8</span> +<span class="normal"> 9</span> +<span class="normal">10</span></pre></div></td> + <td class="code"> + <div class="source"> + <pre><span></span><span class="p">(</span><span class="kd">ns </span><span class="nv">test.core</span> + <span class="p">(</span><span class="ss">:require</span> <span class="p">[</span><span class="nv">reagent.core</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">cursor</span><span class="p">]]</span> + <span class="p">[</span><span class="nv">react-bootstrap</span> <span class="ss">:refer</span> <span class="p">[</span><span class="nv">Button</span><span class="p">]]))</span> + +<span class="p">(</span><span class="nf">defvisr</span> <span class="nv">Counter</span> + <span class="p">(</span><span class="nf">elaborate-fn</span> <span class="p">[{</span><span class="ss">:keys</span> <span class="p">[</span><span class="nv">count</span><span class="p">]}]</span> <span class="nv">count</span><span class="p">)</span> + <span class="p">(</span><span class="nf">render</span> <span class="p">[</span><span class="nv">this</span><span class="p">]</span> + <span class="p">(</span><span class="k">let </span><span class="p">[</span><span class="nb">count </span><span class="p">(</span><span class="nf">cursor</span> <span class="nv">this</span> <span class="p">[</span><span class="ss">:count</span><span class="p">])]</span> + <span class="p">(</span><span class="nb">when-not </span><span class="o">@</span><span class="nb">count </span><span class="p">(</span><span class="nf">reset!</span> <span class="nb">count </span><span class="mi">0</span><span class="p">))</span> + <span class="p">[</span><span class="ss">:&gt;</span> <span class="nv">Button</span> <span class="p">{</span><span class="ss">:on-click</span> <span class="o">#</span><span class="p">(</span><span class="nf">swap!</span> <span class="nb">count </span><span class="nv">inc</span><span class="p">)}</span> <span class="o">@</span><span class="nv">count</span><span class="p">])))</span> +</pre></div> +</td></tr></tbody></table> +</div> + +<p>Here is the VISr in action:</p> + +<div class="figure"><img src="/img/intro-visr/full-count.png" alt="Full Count Example" /> + <p class="caption">Full Count Example</p></div> + +<p>That&rsquo;s all there is to it. From here, you can go to <a href="https://visr.pl">visr.pl</a> to make your own programs using VISr. You can also <a href="https://study.visr.pl">take this survey</a>, which contains more advanced example uses for VISr. If you find any bugs or want to contribute, you can also head to <a href="https://github.com/LeifAndersen/interactive-syntax-clojure">the visr project page</a>.</p> + +<p>Thanks for reading, happy coding!</p> \ No newline at end of file diff --git a/blog/feeds/windows.atom.xml b/blog/feeds/windows.atom.xml new file mode 100644 index 00000000..c0f2c1fd --- /dev/null +++ b/blog/feeds/windows.atom.xml @@ -0,0 +1,37 @@ + + + PRL Blog: Posts tagged 'windows' + + + urn:http-prl-ccs-neu-edu:-blog-tags-windows-html + 2017-05-26T17:00:28Z + + Racket 6.9 and Windows 10 Creators Update + + urn:http-prl-ccs-neu-edu:-blog-2017-05-26-racket-6-9-and-windows-10-creators-update + 2017-05-26T17:00:28Z + 2017-05-26T17:00:28Z + + Leif Andersen + +<p><a href="http://racket-lang.org/">Racket</a> 6.9 was released in April and it has been smooth sailing for many people. However, some people using the <a href="https://blogs.windows.com/windowsexperience/2017/04/11/whats-new-in-the-windows-10-creators-update/">Windows 10 Creators Update</a> have been experiencing <a href="https://github.com/racket/racket/issues/1671">crashes</a>, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.</p> + +<p>The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.</p> + +<p>A VirtualBox image with Racket preinstalled can be downloaded here:</p> + +<ul> + <li><a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova</a></li></ul> + +<p>The username and password for this machine are both <code>racket</code>.</p> +<!-- more--> + +<ol> + <li> + <p>The first thing you need to install is virtualization software. In principle it doesn&rsquo;t matter what you install, but for this tutorial, we will use <a href="https://www.virtualbox.org/">VirtualBox</a>. Go to their <a href="https://www.virtualbox.org/wiki/Downloads">downloads</a> page and download and install the version for your platform (most likely Windows).</p></li> + <li> + <p>Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which <a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">you can download here</a>. The rest of this tutorial will assume you are using this image.</p></li> + <li> + <p>Start up VirtualBox and import the virtual machine. You can do this by clicking on <code>File -&gt; Import Appliance</code>. In the dialog, select the image you downloaded and hit <code>Continue</code>. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click <code>Import</code>.</p></li> + <li> + <p>After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit <code>Start</code>. Once started up, you will see DrRacket and Firefox on the VM&rsquo;s desktop.</p></li></ol> \ No newline at end of file diff --git a/blog/feeds/windows.rss.xml b/blog/feeds/windows.rss.xml new file mode 100644 index 00000000..a431b565 --- /dev/null +++ b/blog/feeds/windows.rss.xml @@ -0,0 +1,37 @@ + + + + PRL Blog: Posts tagged 'windows' + PRL Blog: Posts tagged 'windows' + http://prl.ccs.neu.edu/blog/tags/windows.html + Fri, 26 May 2017 17:00:28 UT + Fri, 26 May 2017 17:00:28 UT + 1800 + + Racket 6.9 and Windows 10 Creators Update + http://prl.ccs.neu.edu/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/?utm_source=windows&utm_medium=RSS + urn:http-prl-ccs-neu-edu:-blog-2017-05-26-racket-6-9-and-windows-10-creators-update + Fri, 26 May 2017 17:00:28 UT + Leif Andersen + +<p><a href="http://racket-lang.org/">Racket</a> 6.9 was released in April and it has been smooth sailing for many people. However, some people using the <a href="https://blogs.windows.com/windowsexperience/2017/04/11/whats-new-in-the-windows-10-creators-update/">Windows 10 Creators Update</a> have been experiencing <a href="https://github.com/racket/racket/issues/1671">crashes</a>, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.</p> + +<p>The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.</p> + +<p>A VirtualBox image with Racket preinstalled can be downloaded here:</p> + +<ul> + <li><a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova</a></li></ul> + +<p>The username and password for this machine are both <code>racket</code>.</p> +<!-- more--> + +<ol> + <li> + <p>The first thing you need to install is virtualization software. In principle it doesn&rsquo;t matter what you install, but for this tutorial, we will use <a href="https://www.virtualbox.org/">VirtualBox</a>. Go to their <a href="https://www.virtualbox.org/wiki/Downloads">downloads</a> page and download and install the version for your platform (most likely Windows).</p></li> + <li> + <p>Once installed, you need to download a virtual image and install Racket on it. We have prepared an image that comes with Racket pre-installed, which <a href="https://github.com/nuprl/website/releases/download/racket69vm/Racket_6_9.ova">you can download here</a>. The rest of this tutorial will assume you are using this image.</p></li> + <li> + <p>Start up VirtualBox and import the virtual machine. You can do this by clicking on <code>File -&gt; Import Appliance</code>. In the dialog, select the image you downloaded and hit <code>Continue</code>. The next window lets you change the specs for your virtual machine. Feel free to make any changes you want, but the defaults work fine for this image. Once you are satisfied click <code>Import</code>.</p></li> + <li> + <p>After import finishes, you should now see your new VM in the list on the left of the VirtualBox manager. Select it and hit <code>Start</code>. Once started up, you will see DrRacket and Firefox on the VM&rsquo;s desktop.</p></li></ol> \ No newline at end of file diff --git a/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png b/blog/img/posts/2016-06-27-tutorial-using-racket-s-ffi/pict.png new file mode 100644 index 0000000000000000000000000000000000000000..0d8a72ee3e0d25d2a4963de4fea256cb3cafb17a GIT binary patch literal 4387 zcmeHL{a2IM9lt=4+JJO*EeHWNj-6*y1F=vdAn4Q~MNO+l1SA2UYQ$iHqIn@SU^_rs zh*MM|<;7W*8Hi+Jw4M;4O--t#q9EZ#2q+~{c?l2%LxAkwBv}0qJlk?ke)&G<-sj%$ z_w$zIf4nuql5001tLn>K6%z}^{s-&_DIXL2W80eJgqci)Ix|1dDYZzuC3hy4!uF|=Cd zk2B!|w-PH>Gf1W>hUQdRd|jKWNLPKPHAC+RMt#jeTszP=v)CPEZ%P2*9L_u;mT31l zfaD!|*{qpvZuNf?2gC=>Cml1F+JO!B4&d$Cey?2JY^%;4G^RyHzZ2jL)-Lr2;CLgj zK419%!hdx?4kb?7$F;>uN&I$2U6{6wSW{rs3Tk}z_+_tQ2W$I@J_VdC6&IV^L22V= zG0)0&t&>A&h^Bbc>ll^Z6PgF>f`Ye+BvJirx+)H%mWvlPaz;&pUcPZJg`%HrUOku` zH$7XE2!OAPdZm%nV0$J~RUtx9sAJTJMH zsb+GY`U2(Pw?d&tDov!XwJzLsNisP`n)5&WyX@gzBdapQxyP|{@LW4v{p-%_>OCJ7 z6MM@fxJE@N-Y^T7nu_wJco!E3AnwZP#MS+HI;HIX3V>HXz<3{O-HIkVeh;6$eM2t( zPTh4gZRVh=sg`Ni@jW~dmqL+ERW4?625z8NYZu8!E*D;k0RqO2rqO$X@H?@e;t%@W7+vDjm^3>eUwYGgY9m1Bx)0FW9fOf&ax02 zdXf-&dJ8!i#8-rNL=aRZ%+P0%y9f3sK~-!EaOO_#ZXInCgfSf0tDAbngzL3$|7{T9=E!ta@%sJmRM;3pQWe^)DA2h1f@<5$8Kbr^u=i4 z;kxSDgB}Y&LZFL$sYvooKl`9+5Tkk-ZVOWQ#_2y!+IM_aiG_Km8+Z1aFpa076762= zFnklY-+DNZoofenk)d|`9BP(dg**22p5_Ooj(&r+L^o^w%qp!F z$9Y&iZ63$}M5-e7=N^gf7#9_lIA+T;xTnv!=Q}$Ciw`w@6iBfsWc2xKZZwOxZnB5d@UH& zj0rDMSyLecpmO&aS>t=?Hk;&P>Ggenpz+Tm2N?nhQFR#8} z(*mB?b~#853l+Syg9+o24&%Uk;my>97a}Gjnm<6rDcj<6YHs-KS;= zEO#yowacIsL?dXC|1|P>KsRc3dfvnLpJ~iO;Q>)uK}7iG-A)NnVV7D~;7iawVK0l+ zUG4pBi!U&YRY!{^e5L7mR`e{hphrqfLHPQ)+sm*I8J8Z4KN3%INp50uIm4sgqm_OT z_|b28PA0*VRoj3?ua;cv-n;9>cnDN8lJ2y*kB#K4!Sp4o^8@ShhihXQ#fq^zg6oR9 z&_u6nL*Ep!IK@1B1r1zRdia`6e+1_x%rtyWA?FKK6-HfR_=QLd|A`8?qlZ*VjL`d( z`-%^@OpxK;HZq40J!}Frlk3?ud$7tT+Vo}-qWCHkl`A) zo|ZZXRRy@vH(JdC0ct$v!SJPT0KO&#fpH|a%r=t53+gUq|8x2Mjo=yv9e;LcBR?OW z3rf}KiE`6tM`?;oD~?lDQ%I>SIJT&Y+djY1x8mv#Z>t5VyHbuBI|a1OUD$RT4fqz# zy-xIN?s81%hbPigV0^0B+C6YeXR}VJ^uvq$>=G_RBa6W1NQ~nv5j>9PCbHV;i26;< zdtnS~vDKn2H6JMFFQ{y)bpw4Tm%p4n)F^x0Y#wm1Y&HMO00tjv1U5}r2vfx;yc`KS zBMLM%*a3HNC-3F-!pc&)R(@vJ_(%sa@Iu6bv{@HXk!o9fKSN*XbCvmYonGd3=)3gG zhz(hut+Xhh8iS3p5PsRn_-&grz$Czj?jjV%GJ5$D6=X>Hrg}vAudS44M~WmtiZr7u zroRK*h0VDeK0*_|^;0}Q*OC`TVD8hHXcUS_QZ)8BFMkju-&#f~$ywiu+7sf;ZHYBy z;8QHVurll2v=h`%StIa~HD@C91ZfK)zVK(Hn8spUYhZ}Efb^)}NMH?k>FIt!5H!+4=NSY-EwV-%ewL_}+ zOqh0jxPi744%}Q?d*_->e~PZE3H*KuvE~@(uxdG0zgjz(*MUoLE$18As)4vKUiD!= zl=9%=06^Ba=CuwPuWkDIpE=k$+>;!2Z1_*UpwWm>ZONc>%lzSwzJiSa(!S!#+QFW8 j9YNpC7yl<9ADW;E6UI_XLc^Q@{6ucty5aI)6TbWpR*Qqf literal 0 HcmV?d00001 diff --git a/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png b/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict.png new file mode 100644 index 0000000000000000000000000000000000000000..b0dfa2d51ddd323dba18b0487530e6645a3439b0 GIT binary patch literal 7156 zcmdT}X;@R&y51NCEomt@fkRV4YgHg=q6LaVtKft}Q4tUlf|76)6e=@F02Oek;7Euv zs3<{{BuW`fWGL1#5#)d*3IxysLLgCKg9HM1t+3-SV!MW{wA!-N?DU&yr>37f+~9tiwB6F{ zK;WHMZ%*yJV}7crq~s(yI?rnB?3GgF@Uvb5m6+3)RgqgUGB%;it#ee^RSo8#4wTo= zVuo#kszP>qAV}8VZUo*@gu7{W&)>&9$ppVd>CjFN_x;6#^QmuR{ zaZ`43ot<{QNE==<^>RW+5|4L?NlKifv^_3;A&z1t>8O$_o zPYmcvra9y(W(#vhuB7hI=y=R?vkN59#c=B-6nyB<{f$h zG7$Lo5flCPV-j+8$N8x@5{<}WB-Nz`K_uV#Bg%)CxJBPUN3I;#D>o{}hel&tw0kkl z+Cw*k)4pDcuf7W{oP`#w(AH^!)WTlW*qAdqF!8Tj-YL1NSZB{!xL`XhAf|ih86B6X zxeJWlJ*w{gRhihC>5m}H3nYXanbJ+(Jcs^7XZKb%N&KyQ;^O7l3xOFNkg3}iAp9i@ zTfPz?*wGNU_n;m4r>)qtIB?>>s}ThM`O!=7w*5we!<^BVG1qotva%EzL=$q|x(u5*~hcVNPKKFskG$iu{ z9w#{5y%{`ypjiEmo}t&Y4FnF)@ki#dISmOmtg8xO@+_x_>z!f<5*8tquzsdI1hHkN z=y~}@gmX9zeSfwD_#Y0>=;%sTPs5o&Gy$t*PD}(*^cWq%9j0eR?N(u!%-1 z3snSyV!h<8pScA~)b3hKT(F}y8r>vhm8SJI`C$n^||k>FT-Ml>3u`po8hlsk;;y|^pB^H z1G=Qc1DBu$1CWeGtN;JW*x%-CBnTr9{s18*g}|dOeEoGF30hd{j|^{QItbY|QJQ~S zuvt8;^VnO@(WolC#i)Y^&`o;Xn`44XT4l=?T-M2o4!?BCLyIts)hTEY#Z z&#wBcqHyO71qvM^uQfGTK1`lj$4HzB!#NGa!RL0NoM3#86SAiqT`g|Ej}P6dO8~0F zUwC7f*;SBHZGqZbP(l=)C5C0J-A+y(97a1`7ou8{7`bhsd+muO&Jtiui=oIgZ+)*E z42*r2#dyIYEKo9UP)T-h#~)|m1cLxW0FD3QgAMp*RJV~FY}Bp8Lmb#BD_duKF~Db} z&O#Yj)bn`}05$jXqB=|5@1MV+h>?BRno!biTXCoh@T~QZP(QDQ*fDKQ0o0>)?}P%o z@w(N2{vbw1mSJ8StfGYUcSabN{V1-k#RwqGPz zf`6-v@r%gVZdXB6&tmqGO__}Lum9tRCnRz85*@3h`DY|;FvpI8hifnbL2yhn+v(WR zt5kps9P8qt9X(rpGTVw|p>CAOs+Qm9KudT{Xj1$I??o z4qNsAB|iI=6haoqNIv9Xi`O>u_u2hO@JY#$Xkv&`-KiOU1$n9ayWvOvFdG|8^kqvp zqlq-y``B0K{M^uMR6-;AoHuE2e!Z~ru(MCVFPza^730Ia2FJ3zw!T&wx?)w^90Pb1 zd<=VfaFS%afW~r_R<)qTB3c8wgc5?LfD^~QmpI(12AwcZsf=9$j4Ag-2Q5NW=RpDh zBL(p`CD_txNC%AT8cj~YgvYe&GEQ*aLOw+_;nt5e`G?|hgB8JZ; zZtpvXvfsWFIUEoMRM-pCqaZ8Z@IOW#d>1>K!Xw=KHK-V$b;WE7R9PFCtZhVozuJS! z%{?G%N(jkMs8Y=bIUETNB^nU1>aAP`PIqWeIWZ2BIp~6kJD*B64Glb_a#b@`RZ0Dp z9nYad-T5|$pOVAWJ(RTVW#co9J}lSf)q{R`H`siYs4hY9LZ4HtYSQDw2+p624!>c`?Vs4g-r>2&Ha_&7_5+nh5k2-@d+~Y z8su4dvie>sATFesXy12>4d61mX2*>l zqw>M~X_+QR_Yb$vr6#Ne5tNq?FTNYL)G&DVRBxd#XLJqy^*C@F;ot${LG|dTo*1LL z^b-YzZEa5vA@j7USc>mEF3B=E9vD^OzcX$>2!5*2fe4M=y((Wp4zb7_+&z(5u?{S?d~@F(1$weOw_rwwTJcLS#(@NS_qN7^?c! zM|p1PAHFh(Y<50b!;aL{@}xs5zqueRQAII9eOhg^)SrGkpo>v4Ue8mw&ZE6XufPaI z1XVg(b{%)>D32;?k)ipQ4xY+}ES&P7=zP(6PMxZ=>`qTcgPYwEPbt?q+GnZAioy+0 zibNAPcTUucM6Sj7sGLx%xW~;a^%7r8NP@ZKLBhyk>a!YERQz6002R#bBgv0mOeq_b z8sadLFSCci2HIrdB~IH*@%-HAj>D`63h5@`vVBvypO-I_wS&mb?YiGr9#r?Q%Z&1` z0i&hX;I-m0>5LXbo+5|PM`cIcrOTZ^t(ZCN!jw0OgXSO7+yQ_hi$k?U2E^=-<`lwf zv1fR^tfnlpGOcPIh<&Bvc#uEH^O}kW6}P?hYcX)r^T>nGX{-dPf?1kX$(q!b?@I|w zK+*NGm?wXI0|5Z!uWTh}w4@@7RoHQgQX0J#IJz$G?)E>^o=R@V#cYz^%;{1ro^9~+ z6zu7%7(k;>oaUu37(X{3Y7`rO@J@lnT6WSPq)KS-04HXW=5SkaA-}Enr)6LJ&F+|$ z2Hnm|pXw#muFoXI{>}>`))pAmK#Kj*exTa3lBd2#1YBUP7_){9XLXZ97w8&K4n; zIPQ`urVag$Gy0asdbr*yfvW%wFk;Bi#A{(OJ{1UhFzT4;*Y@t5uN>KDe13*ZKPZ#& zsfp+2T|f*1(i9RkSZ)`;ZtVlbJ|PPbVZ!(dAJ#@ln{UQj^;}5e&`d2wZ!z@(eReaE z(29nokE<^TS-ZeX@$~IJIM)@h+3YCBU+@9ZdLCw0dIBq$joVZ=3JS|YVu{r^ISgA% zEN;kh01s0v;E3x*j_1fg8qa0jvStV9Z2lmw{2iPV&bg*ez=x9MrqA+k<}^ zjwR?OmltGTIqD;FU|g$|XvdPJ_}_i+{+er~QYR&=^_t(AQLBUs5TOJ|&@roQYQoHZ z1j8((QB&_;`{ZY1^OOd`vIR`DLwyG_{EunX)$ycrQD=yD7j81m6Z;4M;JLjQHn#o1 zG%JNeFQ9DFKyC6E)uD2_%-I7~pEw~o5|i-9kmbWKsTKS7;&di%w$1QQ(Hm|nWSUj3 z0GsE|_RsKd)qNZFJJ`U8;5V)B-Wmxx+WMhB#ORV)IExT!dE18h;8gsZZ9jaJ^CpyU zv>g8T#m7%oRxK)j!x3C*<1|%frhFac#rp9J$}#)Fs3+bq8~^*Zp>!+Lp`0B;E9hKD z<(~kR9Yt)MFWWIS6UO=qt=wf->)R$_PB+(7-wNoO?C;D4Z*yV)={H1irM^u08}WWH zwB1T&hsQG^`S@p#y`9t&;pmE4u8J}dt7*L^-vL}5ENTcd%P(tZxgB@%_r$#6qG|vg ze6AE-wwyMj?+h42jX#e3C^W8hp2~u4kuqqvzX{z&{+qG_T9W* zgGJ?%@hlo`HXtElYs;a?R4|EaWzU`b+zhXk{-4G{{>Ejnwi?#QC*gi`zy2YcS|w1} zkBuZ~5wT~mY4{XOJo#ZYC!lNj$bb!KH8{lJ`u+g0NY=GLp@PBYtckxb~5d7j1&1+QnUJ%V$SXR~knLI5{yVE&-$J*I$sf_p$pti}6}iLe>C=XH|@Adh(L@Oa=qfFEnY>WbV2_e*r5>3hsew zlhw)@*tib}*{`oN4;;D&22)`;G9%InYnlD`@oWt4Bp0UZ^2uh`2Os)a=70(&-T?0w zpZh$iya_Y2{^fHk$jWCYh!o4MkDMmLc9QE|P83*xUiks`U|~#0r9F11KnE8%MscGp zvx#d(Jw=(hupOv;${huryGdfw3(K`uU@w|&aT^WmUtUg3+G{bm@OYXW@>@KNT{a!D z-}~6|p;pz7K&8;it{f*}^kcb_k}DT;ty^N%uI&`qlFH^Oje_q}B;;+X6QATfljNN` z(~UqS@)hde4BOr&glpwO>`09!$tnskm`zl?yHs@=I8(+C^Aka_1tc|n8F^3cx{Xl;U|r947>tN2~kCJdgO%H!#7S%=TjZR4d^4G_S&Eok{>w*9TE3z0muU2wmuCc2c(S2Z(o|9+Oe+P~^?g5Ui zc?Kk!-MSu>Wmp36MaEJSd{Z($M5`=R9Gh49XlSQ#`bn@r3X&%?(Os0%UVMJJM3R*w z{;-qk*aZ~o`&$rGK*!)pcQIX6$IB}ej3?B%jd7^2{?t`OQubMnCpr))q{X+C;2hh~ zyKqI4-Pp9qRPU4vSYY}tLaU?~tH&P!ZEX{kol_=wI(sn%%snwKQi5`#%unrxT-lq^ z5r9R-eg%c!*SSi_>iB`}>R0FY4NSrP4-OMdqF?OaEPkIlk2<-%ZPLm`PO!Bv#6ZeaaG>knxsU39v4 z6=$@7CT~G!K@W10RfFZUi9X40l|o$uQO{yJwqKNxTXd44`z&XY$M*ZS)n5z!nmlM+u~vECs2eR@b~&Ic6&*OJQP`k-?s95n2^cg_ zj6}tUNPFxpGCrP%B-vS?%st2PYZGtEj_&9=I5yGLnJ#1vt=CwqV)se2FEeb@AZ4E? zRlja|8FA}%PE@3(&JY#7j8I37y-VNKqy7b++I)?4m zsqWN1hdfv}G;q?eY~s-g=Y-20HJ?oA+w;9sR&ZK+ytN5l>Yvy!n$H{YD_7l~r(Euz z*iQlE5b~FDExY6UsbMj(fs?cI!!XZiPXTzRmva9=2dnx>+{dX30S0!0;5dP?eQ+$V z>J_?BSU(5f$oBymW{BSH$qD z5yIeDHzP%_RO(u%Emfc;EUy-_%Ghz;!_4um%ywevbx~E3DSW;?cse5civ5#ez$N1u kNPEBpd-^|rKbSPqYY~;wc=f-5U-OVH&aO_p_4K3v17viNasU7T literal 0 HcmV?d00001 diff --git a/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png b/blog/img/posts/2016-06-29-tutorial-racket-ffi-part-2/pict_2.png new file mode 100644 index 0000000000000000000000000000000000000000..1a936485bdab9c38b20216687672c0ffca07f8f1 GIT binary patch literal 4520 zcmeHL{Zms{8orcjt9;pRD{K)_(V@0WA!V}yLZWB~EfuXFiv&W@)S^Th5kr8Gq_T_n z0c4d{G)B@zTNDUUAVdjC6fmh`f(|4xAU8!wA|w#t65`EG!X|ZgcK?I^;tzM`%$;-2 zyz`v*eV+Gu&+JM`@>}}RQUCz_wr|_I8vx$&^?dIxcK^A2YAyf(-Xm|{x_NIQXL6WU z(UzSipFz~09Q3VQ^D#;=^Y+ujzKcKf-Zt>t>aEJN2e)mV`RDuScMr$!E!|hqwVjz0O0(8 zFMhM(HNbChc!R?qTo)&_62>$KnxV|5=YV{}aE|0>L+C)sR1mRA;Ifx$Oz%vL{b|C8 z&%2Mcj8JA>R)}FmfMgS=X^XcJ^R! z5zmSpl3y=hh#K%k=bY#?5X_~K9aUcn$ z#kRxFhW1Yt4OyYyKdVWRQsJH^jX{c#Z@}GX>}^9ZiIJ<=Fkq&hnHv>aok#^rwx1-X zjv8xwq1h$E?oq0T*EwC7_NcjwW%4SNj04$B4m9zoGZ}Ta&OjJ&TxM@jnO{u2C88?G zn9;sivM#Na`3{!=tq!h1EZ4HDJ~$-3xIjWn8^2x9t1ZdQO?b;m&l7ZfQy6@RHhZ;dTguGw)_D;_rUb4gs$aoogRdjpAx7m zJ(*<|SJEQKFfT^7kQnnjb%&SU&4$vFL_!r`0%?i3Ph1@6niHf3fg$F!rA0t5}vU?C1di zs1wt1`txh-*D8xtLBz+g*mk8Q!)z&dZe|zO#=m8puEZ=xG zKpVqs7+t#J?(_0fy~8UcI(uOdO}tRdy%=lcs%j(>Ls$;R_4m}lsV5c66^J{ zCIJ{LTgSgi?sz$LWlnSw#6unKeB9yuD9_&Zh<0_(8RoBt59y@B)?(T#mm$+$FLpF@ zsqqY!C+AMjx0kI>BMnE%g7TGM-4o>Vxj5T-iKNM}k0NdqMv#^8xUntAd4wGfQj;(d zGd;Hc+WS&6XW{goOn#(IZfsjw{tL_T%(7t$8CTVJy8f->(=#r(Y!Wz7wyr83GnM!F zi+MT?8Getyl2{>7i+0{IgDZ+0Q+8M%gSf;Y9rT3AaB9QuB5@A|T7%wGqAWoGljzh&5zWs5jCfb|OnOgYmz&Qw6&|XvNHC6<4iJ{kU$pnQX~cnKj4w2~(xzS**Uu zhj(3IfwNG&$>{7X0jFNFbf{Bq;k&oqO!V2pJew{xXp(jzzp|Z!5J6q?$RnT~A8fD^ z#v+@v+(uA-%+h!LRR>#ID6@)ta3()u#_U~#^JaCb+8p8S{Ae;y{u3E53uxBT5F0{8 z(Dmi2K1<>x!WVYV8SmSWE@XHe{VN1d{}OK*MxLT=vuPpOl$SOW&NPUV^1=O5*SYJ$ zO$n`rJXTF6*i0r%H`cML{{>Q+9~ami7c$($>JJwVpYZDIHBO{VzX~C~R*dJ*&zw*e zJq%Rih61_*(jUi4Z@MYuOlY($h{U7jN*7ZXX%kL;hg zl~g?D--ul_eV6zuxoknG?$AjzCwcLiv1reW|BCSizNfnvLp%TAF3RJC-Z-2$PVbH5 z{X?Ap%a#)-V~v9k&H_Mrucw>wOdU=%X6+{_=uPhe$Aa1g_cD4vTMFEJ*typ0e*+3< ZC@)xX literal 0 HcmV?d00001 diff --git a/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png b/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict.png new file mode 100644 index 0000000000000000000000000000000000000000..2f596378e8d19836895e3549c1e63b86cac63a6b GIT binary patch literal 3615 zcmeHKYgdz37CtdU#F%&+s2Z6%Qn6v-K$eY|R%*D(B|r?&nJ~a;T(#D>AZFpziB}Re z3{?gPc`FuLB|^ZpkO(?js2~AmtYQTX3XC!#ULacHpafwgiOde?(7(_x{lHoeXYcdu zXYaGmK5xE|Nu#-J9vdOVjg5&+LTCoZ{pZPom9KI~=fe76Y-HF+s;{5_r|!r~YVoy~ z!$rSVF@N+sd2Z{$x`4dAJf4bMUAV?~g+Z>^R_4pPp!;Hl@6kmiv#RS3#x72|)$x8+ zXUK=HKYV^q{>tHZ?}xyhAG~jC%h_Xn-X&a()Yg4lht?ouJKcg%B#VjWHGMbsFY}eT z!N?e!&~>-pc|T3tl7iGbuUz(U#>_x@92WZRcN2?qs|+q5CK8<(E&P?1ftOb;Lh9=Z zgnSSEvfwV$&sdMA_quneAB>vCCpW#haH+eJn%L6zEz@{|&q1MgtEfG$*-g!3TB>9s zqJRGRXHApJ1FtW>e*Mz9uX(keq5h;|@=XAERtRkA!>*n7o5u#N#e>%VL2FNH#`coR zoRE?I4CP8T6UizCoNx*$5(th0%q3U?IF?{3;8_^^95$g)Z7V{wKWApS5Gm(E|L~I+ z%>jRlpBD?2#cvjgKt-Bc>{qSI1rjFeOoZNArG2=CW+4N?6@X7A;q@ngQWjh^l7b-BDJzy*5`XZLY$SM<}=qenxQEfs#Ut)UYOg%%y!}MjL6ecDWJ@ zvOz~=uEzm-b!LavT3l>;VRkxI2d?&~W^@Vnk6a)3@F{jnZn{6gMLVw0hs=(pgF#K> z*X-r5PX};~5$+gI-s99pG>z}*49Zz8o?I)QTqkZ8iT77BQVw^LP}hql*FFle+h^PD z-`njA?OnqZdP`4?p~J3`aoZZZJLf7cflICM3L{AssY11Z2>r$&#@42tn zotW!s$8fa}{RQ7QM)2*u?m5xobPhS4k|~1WN8@sydLeOm`|S9eTdxOpSgix>0ePBx zQiy|`R)4NBfz*3ujGSLZO+0JQxoScoapctQ#GRL|v$)_aeoqoP^53TLhG0EIRfHR% z_MqM%GbB(bkPvy)e6G#jE@778+N|96Jb=zf=2;r2?I9EFf%`N|w3Kv@VhB5g5Qnw4 z=h$H#X^S~~amWIqN~KAh3aNffvda?u=-qzFpR4pCBDqn`QzEaC&JB6IP0sJB6dLPE zjDUgzQzG-DaiXe6nb<)@9Nz8NGv#1AjnllHuG{COXN(aE(p%o`+xgsDF&Rgn8Dv>; zl-fluadNgcdlSv-{pnr?&<=>c3zJ*C(se#ZP3Ra=S7l)joTOPQADpj9DCZvwqtvhP z)if?^2;UtqVV)yW=iKma|2Uba*iAz74YE@}b`#e^6QR_O^wF%dWXS3o!fzx?n0*^? zp1Ff;KbX5OvhilYayh>NM&q~B-u2-(KBQUKu}%e0C+UR7D@4e=blsL%39~T)bM3n2 zE$OZh8mowRak}nFAIaiiOA3I@lbDx*nM1Rp$e5_k z)*gQ^iMOZ_ALTjl?ue%p%W(~A*>e6vDA7CQ?AAAg&l;{2)_zLNo?zZ3VeTa_)lsE( z!A2>i*nne|{nJ}=HksGU#6s(j2GF%3dd=@h$^MkCJN$~z%_OhKM(|eo=rz$~@R!zy z&w7x^Q@Ov>;GI>SfZaWia9@8Y!P0GGmF z($;{dnNk3| zySdNPxpi2X08UGj$mdRPo(G%D6M5J0OWxK2oAagAqC@UZaK8xiT#5r5+gk_N`U{A& z89tvbcc%`!hU~?G`iDuf=CxvRrp^7@N3LrV7mHJ7N8Hy0m1VAMY->Q8aF zx>HLlpIMh&{ms^OZBw64NDVvgE{1ny3)Hr-g?zsMvhZ`(<4*Nx(5Sg8x^X^IG{fDU z`U&u~rbUpD-2dor2sxD7XOB`N^F5Gd_kraM{EN$dWTWB+-#?FgO$M6bhaZYvFO4*W HD-Ql2j3>B= literal 0 HcmV?d00001 diff --git a/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png b/blog/img/posts/2016-07-11-tutorial-racket-ffi-part-3/pict_2.png new file mode 100644 index 0000000000000000000000000000000000000000..2f596378e8d19836895e3549c1e63b86cac63a6b GIT binary patch literal 3615 zcmeHKYgdz37CtdU#F%&+s2Z6%Qn6v-K$eY|R%*D(B|r?&nJ~a;T(#D>AZFpziB}Re z3{?gPc`FuLB|^ZpkO(?js2~AmtYQTX3XC!#ULacHpafwgiOde?(7(_x{lHoeXYcdu zXYaGmK5xE|Nu#-J9vdOVjg5&+LTCoZ{pZPom9KI~=fe76Y-HF+s;{5_r|!r~YVoy~ z!$rSVF@N+sd2Z{$x`4dAJf4bMUAV?~g+Z>^R_4pPp!;Hl@6kmiv#RS3#x72|)$x8+ zXUK=HKYV^q{>tHZ?}xyhAG~jC%h_Xn-X&a()Yg4lht?ouJKcg%B#VjWHGMbsFY}eT z!N?e!&~>-pc|T3tl7iGbuUz(U#>_x@92WZRcN2?qs|+q5CK8<(E&P?1ftOb;Lh9=Z zgnSSEvfwV$&sdMA_quneAB>vCCpW#haH+eJn%L6zEz@{|&q1MgtEfG$*-g!3TB>9s zqJRGRXHApJ1FtW>e*Mz9uX(keq5h;|@=XAERtRkA!>*n7o5u#N#e>%VL2FNH#`coR zoRE?I4CP8T6UizCoNx*$5(th0%q3U?IF?{3;8_^^95$g)Z7V{wKWApS5Gm(E|L~I+ z%>jRlpBD?2#cvjgKt-Bc>{qSI1rjFeOoZNArG2=CW+4N?6@X7A;q@ngQWjh^l7b-BDJzy*5`XZLY$SM<}=qenxQEfs#Ut)UYOg%%y!}MjL6ecDWJ@ zvOz~=uEzm-b!LavT3l>;VRkxI2d?&~W^@Vnk6a)3@F{jnZn{6gMLVw0hs=(pgF#K> z*X-r5PX};~5$+gI-s99pG>z}*49Zz8o?I)QTqkZ8iT77BQVw^LP}hql*FFle+h^PD z-`njA?OnqZdP`4?p~J3`aoZZZJLf7cflICM3L{AssY11Z2>r$&#@42tn zotW!s$8fa}{RQ7QM)2*u?m5xobPhS4k|~1WN8@sydLeOm`|S9eTdxOpSgix>0ePBx zQiy|`R)4NBfz*3ujGSLZO+0JQxoScoapctQ#GRL|v$)_aeoqoP^53TLhG0EIRfHR% z_MqM%GbB(bkPvy)e6G#jE@778+N|96Jb=zf=2;r2?I9EFf%`N|w3Kv@VhB5g5Qnw4 z=h$H#X^S~~amWIqN~KAh3aNffvda?u=-qzFpR4pCBDqn`QzEaC&JB6IP0sJB6dLPE zjDUgzQzG-DaiXe6nb<)@9Nz8NGv#1AjnllHuG{COXN(aE(p%o`+xgsDF&Rgn8Dv>; zl-fluadNgcdlSv-{pnr?&<=>c3zJ*C(se#ZP3Ra=S7l)joTOPQADpj9DCZvwqtvhP z)if?^2;UtqVV)yW=iKma|2Uba*iAz74YE@}b`#e^6QR_O^wF%dWXS3o!fzx?n0*^? zp1Ff;KbX5OvhilYayh>NM&q~B-u2-(KBQUKu}%e0C+UR7D@4e=blsL%39~T)bM3n2 zE$OZh8mowRak}nFAIaiiOA3I@lbDx*nM1Rp$e5_k z)*gQ^iMOZ_ALTjl?ue%p%W(~A*>e6vDA7CQ?AAAg&l;{2)_zLNo?zZ3VeTa_)lsE( z!A2>i*nne|{nJ}=HksGU#6s(j2GF%3dd=@h$^MkCJN$~z%_OhKM(|eo=rz$~@R!zy z&w7x^Q@Ov>;GI>SfZaWia9@8Y!P0GGmF z($;{dnNk3| zySdNPxpi2X08UGj$mdRPo(G%D6M5J0OWxK2oAagAqC@UZaK8xiT#5r5+gk_N`U{A& z89tvbcc%`!hU~?G`iDuf=CxvRrp^7@N3LrV7mHJ7N8Hy0m1VAMY->Q8aF zx>HLlpIMh&{ms^OZBw64NDVvgE{1ny3)Hr-g?zsMvhZ`(<4*Nx(5Sg8x^X^IG{fDU z`U&u~rbUpD-2dor2sxD7XOB`N^F5Gd_kraM{EN$dWTWB+-#?FgO$M6bhaZYvFO4*W HD-Ql2j3>B= literal 0 HcmV?d00001 diff --git a/blog/index-2.html b/blog/index-2.html new file mode 100644 index 00000000..8a9aa372 --- /dev/null +++ b/blog/index-2.html @@ -0,0 +1,260 @@ + + + + + + PRL Blog (page 2) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + + + + +
+
+

PLISS: Learn About PL Implementation in a Castle

+

+ :: event, lectures, castle, language implementation

+

By: Alexi Turcotte

+
+ +

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

+ +

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

+ +
+ +
+
+

On-Stack Replacement

+

+ ::

+

By: Ming-Ho Yee

+
+ +

Last semester, I took a course where the final project was to write a survey paper on “a topic in the intersection between computer systems and your area.” So I wrote about on-stack replacement.

+ +
+ + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/index-3.html b/blog/index-3.html new file mode 100644 index 00000000..783ec494 --- /dev/null +++ b/blog/index-3.html @@ -0,0 +1,256 @@ + + + + + + PRL Blog (page 3) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Defining Local Bindings in Turnstile Languages

+

+ :: turnstile, tutorial, language, dsl

+

By: Sam Caldwell

+
+ +

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

+ +

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

+ +

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

+ +

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

+ +
+ + + + + + + + +
+
+

Final Algebra Semantics is Observational Equivalence

+

+ :: category theory, math, final encoding, observational equivalence

+

By: Max New

+
+ +

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

+ +

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/index-4.html b/blog/index-4.html new file mode 100644 index 00000000..17c58336 --- /dev/null +++ b/blog/index-4.html @@ -0,0 +1,272 @@ + + + + + + PRL Blog (page 4) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + + + +
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ +
+
+
+

Gradual Typing Across the Spectrum, part II

+

+ :: gradual typing, PI meeting

+

By: Ben Greenman

+
+ +

Last week, Northeastern hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:

+ +

http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html

+ +

A common thread among the talks was the question: how to convert a research idea into a tool for software developers?

+ +
+
+
+

Reviews and author responses: we should stop asking for 500-word responses

+

+ ::

+

By: Gabriel Scherer

+
+ +

This year I reviewed many ICFP submissions, and got to be on the receiving end of equally many author responses (also sometimes called, somewhat combatively, rebuttals). I found that there was a large difference between the official written advice on author responses and what I, as a reviewer reading the responses, found effective. In particular, I now believe that limiting yourself to 500 words should strongly be avoided — we should even stop giving that advice.

+ +
+ +
+
+

Continuations

+

+ :: history

+

By: Ben Greenman

+
+ +

From the PRL archives:

+ +
+

It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. — Continuations by Alan Nall, Indiana University, 1983

+ +
+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/index-5.html b/blog/index-5.html new file mode 100644 index 00000000..fdce0f82 --- /dev/null +++ b/blog/index-5.html @@ -0,0 +1,292 @@ + + + + + + PRL Blog (page 5) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Report: PLISS 2017

+

+ :: pliss, event

+

By: Ming-Ho Yee

+
+ +

Two weeks ago, I attended the first Programming Language Implementation Summer School, held in beautiful Bertinoro, Italy.

+ +

The goal of PLISS was “to prepare early graduate students and advanced undergraduates for research in the field,” and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.

+ +
+
+
+

Syntactic parametricity strikes again

+

+ ::

+

By: Gabriel Scherer, Li-Yao Xia

+
+ +

In this blog post, reporting on a collaboration with Li-Yao Xia, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely “syntactic” way, by looking at the structure of canonical derivations. More precisely, I prove that \( +\newcommand{\List}[1]{\mathsf{List}~#1} +\newcommand{\Fin}[1]{\mathsf{Fin}~#1} +\newcommand{\Nat}[1]{\mathbb{N}} +\newcommand{\rule}[2]{\frac{\displaystyle \array{#1}}{\displaystyle #2}} +\newcommand{\judge}[2]{{#1} \vdash {#2}} +\newcommand{\emptyrule}[1]{\begin{array}{c}\\[-1em] #1 \end{array}} + ∀α. \List α → \List \alpha +\) is isomorphic to \( + Π(n:\Nat{}). \List{(\Fin{n})} +\) where \(\Fin{n}\) is the type of integers smaller than \(n\), corresponding to the set \(\{0, 1, \dots, n-1\}\).

+ +
+
+
+

Racket 6.9 and Windows 10 Creators Update

+

+ :: racket, windows, bugs

+

By: Leif Andersen

+
+ +

Racket 6.9 was released in April and it has been smooth sailing for many people. However, some people using the Windows 10 Creators Update have been experiencing crashes, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.

+ +

The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.

+ +

A VirtualBox image with Racket preinstalled can be downloaded here:

+ + + +

The username and password for this machine are both racket.

+ +
+
+
+

Programming Language Conference in Russia

+

+ ::

+

By: Artem Pelenitsyn

+
+ +

In April 3—5 I took part into a Russian conference exclusively devoted to programming languages: Programming Languages and Compilers (Google.Translated version of the site). I was a member of organizing committee and had a paper there.

+ +

This is the first conference in Russia highly focused on our area of PL. At least for the last several decades (I believe, there were conferences of the kind back in USSR). The conference was devoted to the memory of prominent Soviet PL-researcher from Rostov-on-Don, Adolf Fuksman who worked on ideas quite similar to what we know as the aspect-oriented programming back in the 70-s.

+ +
+ + + +
+
+

Rank Polymorphism

+

+ :: array language

+

By: Justin Slepak

+
+ +

Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let’s just call it lerp) for scalars:

+ +
(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))
+ +

The number marks on each argument indicate the expected “rank” of the argument: how many dimensions it should have. In this case, each one is marked 0, indicating a scalar (i.e., 0-dimensional) argument. The function is usable as-is for

+ +
    +
  • +

    α-blending two RGB pixels

    • +
  • +

    dimming or brightening an image

    • +
  • +

    fade transition between video scenes

+ +
+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/index-6.html b/blog/index-6.html new file mode 100644 index 00000000..fc30197b --- /dev/null +++ b/blog/index-6.html @@ -0,0 +1,259 @@ + + + + + + PRL Blog (page 6) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

PRL at SNAPL’17

+

+ ::

+

By: Gabriel Scherer

+
+ +

PRL recently produced three papers for the SNAPL conference.

+ + + +
+ + + + + + + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/index-7.html b/blog/index-7.html new file mode 100644 index 00000000..0136d9b5 --- /dev/null +++ b/blog/index-7.html @@ -0,0 +1,258 @@ + + + + + + PRL Blog (page 7) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Bullets are good for your Coq proofs

+

+ :: coq

+

By: Gabriel Scherer

+
+ +

I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.

+ +

Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.

+ +
+ + + + +
+
+

Fall 2016 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell

+
+ +

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

+ +
    +
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. +
  2. they represent a significant increase in the reasoning power of types over programs.
+ +
+
+
+

Measuring the submission/review balance

+

+ ::

+

By: Gabriel Scherer

+
+ +

How do researchers know whether they are doing “enough” or “too many” reviews? A measurable goal is to be review-neutral: to have demanded, through our submissions, as many reviews as we have produced as reviewers.

+ +
+ +
+
+

SRC-submissions

+

+ ::

+

By: Gabriel Scherer

+
+ +

Max New, Daniel Patterson and Ben Greenman recently wrote three two-page abstracts on what they are working on right now. Come have a look — and any feedback is welcome!

+ +
+
+
+

Understanding Constructive Galois Connections

+

+ :: icfp, galois connection, adjunction, category theory, math

+

By: Max New

+
+ +

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/index-8.html b/blog/index-8.html new file mode 100644 index 00000000..1da975be --- /dev/null +++ b/blog/index-8.html @@ -0,0 +1,285 @@ + + + + + + PRL Blog (page 8) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Beta Reduction (Part 1)

+

+ :: lambda, calculus, beta, reduction, semantics

+

By: Milo Davis

+
+ +

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

+ +
+ +
+
+

History of Actors

+

+ :: history

+

By: Tony Garnock-Jones

+
+ +

Christos Dimoulas is currently teaching a “History of Programming Languages” class at Harvard. The class is, as Christos writes, “definitely not about this”; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.

+ +

On Monday, I presented “the History of Actors” for the class. I’ve made the written-out talk notes and an annotated bibliography available here.

+ +
+
+
+

Emacs daemon for fast editor startup

+

+ :: System Administration, Emacs

+

By: Gabriel Scherer

+
+ +

In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs’ bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors — on each git commit for example, or when I use a Firefox extension to edit a textarea content in a proper editor.

+ +

In this blog post, I describe how to use emacsclient to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.

+ +
+ + + + +
+
+

Tutorial: Racket FFI, part 3

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +here +and part 2 +here.

+ +

In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.

+ +
+
+
+

Tutorial: Racket FFI, Part 2

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 2 of my tutorial on using the Racket FFI. If you haven’t read +part 1 yet, you can find it +here. +Update: part 3 is also now available +here.

+ +

Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/index-9.html b/blog/index-9.html new file mode 100644 index 00000000..d58d8aea --- /dev/null +++ b/blog/index-9.html @@ -0,0 +1,257 @@ + + + + + + PRL Blog (page 9) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + +
+
+

Tutorial: Using Racket’s FFI

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

Update: this post is now part of a series. Part 2 is +here +and part 3 is +here.

+ +

I’ve seen several people ask for a tutorial on Racket’s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +documentation +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.

+ +

With that in mind, this blog post will provide a step-by-step tutorial +for Racket’s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.

+ +
+ + +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+
+
+

Gradual Typing Across the Spectrum

+

+ :: gradual typing, PI meeting

+

By: Asumu Takikawa

+
+ +
+

Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. — Matthias Felleisen

+ +

Yesterday we hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found here). In case you aren’t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.

+ +

Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing—the ideas that cross language and platform barriers.

+ +
+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/index.html b/blog/index.html new file mode 100644 index 00000000..fe56c373 --- /dev/null +++ b/blog/index.html @@ -0,0 +1,289 @@ + + + + + + PRL Blog + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ + + + +
+
+

Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript)

+

+ :: visr, clojure, clojurescript, interactive syntax

+

By: Leif Andersen

+
+ +

+

+ +

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

+ +
Graphical network embedded in text +

Graphical network embedded in text

+ +

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

+ +
    +
  • version control, such as git, works with interactive-syntax;
    • +
  • programs using interactive-syntax can be written and edited with your favorite text editor or IDE;
    • +
  • cut/copy/paste works with interactive-syntax using your operating system’s native clipboard;
    • +
  • code analysis tools, like diff and refactor, still work with interactive-syntax; and
    • +
  • you can use interactive-syntax in any language or environment that supports language-oriented programming.
+ +

To learn more about interactive-syntax, watch this video or read the accompanying paper.

+ + + +

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

+ +

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

+ +
+ + + + + + +
+
+

Four Kinds of Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the first and second parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.

+ +

In this third and final part of my blog series, I’d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.

+ +

This is a “bonus” blog post, because I’m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.

+ +
+
+
+

Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the previous post of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: is R lexically or dynamically scoped?

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/sitemap.txt b/blog/sitemap.txt new file mode 100644 index 00000000..c9477490 --- /dev/null +++ b/blog/sitemap.txt @@ -0,0 +1,87 @@ +http://prl.ccs.neu.edu/blog/2016/10/31/meaningful-distinctions/ +http://prl.ccs.neu.edu/blog/2017/05/09/no-good-answers-gradually-typed-object-oriented-languages/ +http://prl.ccs.neu.edu/blog/2017/04/28/what-is-soft-typing/ +http://prl.ccs.neu.edu/blog/2019/05/11/-conversational-concurrency-cross-post-https-eighty-twenty-org-2018-01-24-conversational-concurrency/ +http://prl.ccs.neu.edu/blog/2017/04/25/prl-at-snapl-17/ +http://prl.ccs.neu.edu/blog/2017/04/17/type-directed-compilation-parts-i-and-ii/ +http://prl.ccs.neu.edu/blog/2017/10/22/monotonicity-types-towards-a-type-system-for-eventual-consistency/ +http://prl.ccs.neu.edu/blog/2016/10/17/emacs-daemon-for-fast-editor-startup/ +http://prl.ccs.neu.edu/blog/2017/05/23/building-a-website-with-scribble/ +http://prl.ccs.neu.edu/blog/2016/11/17/src-submissions/ +http://prl.ccs.neu.edu/blog/2020/01/15/the-typed-racket-optimizer-vs-transient/ +http://prl.ccs.neu.edu/blog/2017/03/10/type-inference-in-stack-based-programming-languages/ +http://prl.ccs.neu.edu/blog/2017/04/20/refinement-types/ +http://prl.ccs.neu.edu/blog/2016/09/15/nepls-on-october-7th-at-northeastern-university/ +http://prl.ccs.neu.edu/blog/2018/11/24/disappearing-code/ +http://prl.ccs.neu.edu/blog/2017/07/19/trees-1973/ +http://prl.ccs.neu.edu/blog/2018/12/11/the-behavior-of-gradual-types-a-user-study/ +http://prl.ccs.neu.edu/blog/2017/06/05/syntactic-parametricity-strikes-again/ +http://prl.ccs.neu.edu/blog/2016/05/18/gradual-typing-across-the-spectrum/ +http://prl.ccs.neu.edu/blog/2019/09/10/four-kinds-of-scoping-in-r/ +http://prl.ccs.neu.edu/blog/2017/03/15/tracing-jits-for-dynamic-languages/ +http://prl.ccs.neu.edu/blog/2017/05/26/racket-6-9-and-windows-10-creators-update/ +http://prl.ccs.neu.edu/blog/2017/08/28/closure-conversion-as-coyoneda/ +http://prl.ccs.neu.edu/blog/2017/06/09/-bridging-the-system-configuration-gap-cross-post-https-aaronweiss-us-posts-2017-06-05-bridging-the-system-configuration-gap-html/ +http://prl.ccs.neu.edu/blog/2017/01/02/fall-2016-pl-junior-retrospective/ +http://prl.ccs.neu.edu/blog/2018/05/08/sampling-gradual-typing-performance/ +http://prl.ccs.neu.edu/blog/2018/10/06/a-spectrum-of-type-soundness-and-performance/ +http://prl.ccs.neu.edu/blog/2019/01/28/on-stack-replacement/ +http://prl.ccs.neu.edu/blog/2017/08/29/-why-am-i-going-to-icfp-2017-cross-post-https-williamjbowman-com-blog-2017-08-29-why-am-i-going-to-icfp-2017/ +http://prl.ccs.neu.edu/blog/2017/02/15/introducing-hopl-2017/ +http://prl.ccs.neu.edu/blog/2017/02/28/linear-types-for-low-level-languages/ +http://prl.ccs.neu.edu/blog/2020/10/15/transient-answers-old-questions/ +http://prl.ccs.neu.edu/blog/2019/10/31/complete-monitors-for-gradual-types/ +http://prl.ccs.neu.edu/blog/2016/05/24/measuring-gc-latencies-in-haskell-ocaml-racket/ +http://prl.ccs.neu.edu/blog/2016/04/29/welcome-to-the-prl-blog/ +http://prl.ccs.neu.edu/blog/2017/08/13/reviews-and-author-responses-we-should-stop-asking-for-500-word-responses/ +http://prl.ccs.neu.edu/blog/2016/07/11/tutorial-racket-ffi-part-3/ +http://prl.ccs.neu.edu/blog/2019/12/12/prl-offsite-2019-retrospective/ +http://prl.ccs.neu.edu/blog/2018/12/02/java-and-migratory-typing/ +http://prl.ccs.neu.edu/blog/2017/01/03/-toward-type-preserving-compilation-of-coq-at-popl17-src-cross-post-https-williamjbowman-com-blog-2017-01-03-toward-type-preserving-compilation-of-coq-at-popl17-src/ +http://prl.ccs.neu.edu/blog/2016/06/27/tutorial-using-racket-s-ffi/ +http://prl.ccs.neu.edu/blog/2018/04/27/the-racket-school-2018-create-your-own-language/ +http://prl.ccs.neu.edu/blog/2017/08/22/gradual-typing-across-the-spectrum-part-ii/ +http://prl.ccs.neu.edu/blog/2016/10/19/history-of-actors/ +http://prl.ccs.neu.edu/blog/2019/02/17/writing-a-paper-with-scribble/ +http://prl.ccs.neu.edu/blog/2016/08/02/tutorial-zero-to-sixty-in-racket/ +http://prl.ccs.neu.edu/blog/2017/05/24/programming-language-conference-in-russia/ +http://prl.ccs.neu.edu/blog/2017/06/05/report-pliss-2017/ +http://prl.ccs.neu.edu/blog/2020/12/23/deep-and-shallow-types/ +http://prl.ccs.neu.edu/blog/2016/08/03/a-few-cores-too-many/ +http://prl.ccs.neu.edu/blog/2020/11/12/transient-for-optional-and-keyword-functions/ +http://prl.ccs.neu.edu/blog/2016/12/17/measuring-the-submission-review-balance/ +http://prl.ccs.neu.edu/blog/2017/02/21/datalog-for-static-analysis/ +http://prl.ccs.neu.edu/blog/2016/11/16/understanding-constructive-galois-connections/ +http://prl.ccs.neu.edu/blog/2016/05/03/nepls-on-may-31st-at-umass-amherst/ +http://prl.ccs.neu.edu/blog/2019/03/09/pliss-learn-about-pl-implementation-in-a-castle/ +http://prl.ccs.neu.edu/blog/2019/09/10/scoping-in-r/ +http://prl.ccs.neu.edu/blog/2018/04/23/-how-to-prove-a-compiler-fully-abstract-cross-post-https-dbp-io-essays-2018-04-19-how-to-prove-a-compiler-fully-abstract-html/ +http://prl.ccs.neu.edu/blog/2016/11/30/-getting-started-in-programming-languages-cross-post-http-jschuster-org-blog-2016-11-29-getting-started-in-programming-languages/ +http://prl.ccs.neu.edu/blog/2018/10/22/defining-local-bindings-in-turnstile-languages/ +http://prl.ccs.neu.edu/blog/2018/11/30/turnstile-mailing-list/ +http://prl.ccs.neu.edu/blog/2019/04/07/forgetful-and-heedful-contracts/ +http://prl.ccs.neu.edu/blog/2017/06/24/quotes-and-stories-from-turing-50/ +http://prl.ccs.neu.edu/blog/2017/02/15/conversational-context-and-concurrency/ +http://prl.ccs.neu.edu/blog/2016/06/13/does-anyone-still-care-about-printed-proceedings-grab-some-at-neu-this-week/ +http://prl.ccs.neu.edu/blog/2017/02/21/bullets-are-good-for-your-coq-proofs/ +http://prl.ccs.neu.edu/blog/2016/06/29/tutorial-racket-ffi-part-2/ +http://prl.ccs.neu.edu/blog/2017/05/15/artifacts-for-semantics/ +http://prl.ccs.neu.edu/blog/2017/03/24/-what-even-is-compiler-correctness-cross-post-https-williamjbowman-com-blog-2017-03-24-what-even-is-compiler-correctness/ +http://prl.ccs.neu.edu/blog/2022/02/16/-r%CC%8C-overview-i-cross-post-https-www-o1o-ch-lab-entry-1309rkp8krzaq1catz4by5gyt719ejoi4qpiabpnmzsx64fke3g4huga3b0qqinc-html/ +http://prl.ccs.neu.edu/blog/2018/04/12/-making-an-ide-plugin-for-drracket-cross-post-https-lang-video-blog-2018-03-21-making-an-ide-plugin-for-drracket/ +http://prl.ccs.neu.edu/blog/2017/09/25/plt-redex-faq/ +http://prl.ccs.neu.edu/blog/2017/09/27/final-algebra-semantics-is-observational-equivalence/ +http://prl.ccs.neu.edu/blog/2017/04/04/top-five-results-of-the-past-50-years-of-programming-languages-research/ +http://prl.ccs.neu.edu/blog/2017/02/28/pliss-oregon-without-the-greek/ +http://prl.ccs.neu.edu/blog/2016/11/02/beta-reduction-part-1/ +http://prl.ccs.neu.edu/blog/2017/05/01/categorical-semantics-for-dynamically-typed-programming-languages/ +http://prl.ccs.neu.edu/blog/2019/09/05/lexical-and-dynamic-scope/ +http://prl.ccs.neu.edu/blog/2017/07/17/continuations/ +http://prl.ccs.neu.edu/blog/2017/06/16/spring-2017-pl-junior-retrospective/ +http://prl.ccs.neu.edu/blog/2016/10/11/compcert-overview/ +http://prl.ccs.neu.edu/blog/2018/01/19/-untyped-programs-don-t-exist-cross-post-https-williamjbowman-com-blog-2018-01-19-untyped-programs-don-t-exist/ +http://prl.ccs.neu.edu/blog/2018/01/17/-how-to-prove-a-compiler-correct-cross-post-https-dbp-io-essays-2018-01-16-how-to-prove-a-compiler-correct-html/ +http://prl.ccs.neu.edu/blog/2017/05/04/rank-polymorphism/ +http://prl.ccs.neu.edu/blog/2017/03/03/plt-redex-mf-apply/ +http://prl.ccs.neu.edu/blog/2016/06/07/icfp-2016-looking-for-student-volunteers/ +http://prl.ccs.neu.edu/blog/2022/01/06/introducing-visual-and-interactive-syntax-realized-visr-for-clojurescript-and-javascript/ diff --git a/blog/tags/1st-blog-post.html b/blog/tags/1st-blog-post.html new file mode 100644 index 00000000..3edeadc5 --- /dev/null +++ b/blog/tags/1st-blog-post.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged '1st blog post' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged 1st blog post

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Aaron-Weiss.html b/blog/tags/Author-Aaron-Weiss.html new file mode 100644 index 00000000..0eed85fe --- /dev/null +++ b/blog/tags/Author-Aaron-Weiss.html @@ -0,0 +1,125 @@ + + + + + + Posts tagged 'Author: Aaron Weiss' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Aaron Weiss

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Alexi-Turcotte.html b/blog/tags/Author-Alexi-Turcotte.html new file mode 100644 index 00000000..89cf1e24 --- /dev/null +++ b/blog/tags/Author-Alexi-Turcotte.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'Author: Alexi Turcotte' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Alexi Turcotte

+ +
+
+

PLISS: Learn About PL Implementation in a Castle

+

+ :: event, lectures, castle, language implementation

+

By: Alexi Turcotte

+
+ +

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

+ +

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Artem-Pelenitsyn.html b/blog/tags/Author-Artem-Pelenitsyn.html new file mode 100644 index 00000000..40e7a95b --- /dev/null +++ b/blog/tags/Author-Artem-Pelenitsyn.html @@ -0,0 +1,145 @@ + + + + + + Posts tagged 'Author: Artem Pelenitsyn' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Artem Pelenitsyn

+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+
+
+

Programming Language Conference in Russia

+

+ ::

+

By: Artem Pelenitsyn

+
+ +

In April 3—5 I took part into a Russian conference exclusively devoted to programming languages: Programming Languages and Compilers (Google.Translated version of the site). I was a member of organizing committee and had a paper there.

+ +

This is the first conference in Russia highly focused on our area of PL. At least for the last several decades (I believe, there were conferences of the kind back in USSR). The conference was devoted to the memory of prominent Soviet PL-researcher from Rostov-on-Don, Adolf Fuksman who worked on ideas quite similar to what we know as the aspect-oriented programming back in the 70-s.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Asumu-Takikawa.html b/blog/tags/Author-Asumu-Takikawa.html new file mode 100644 index 00000000..ab2ff6ed --- /dev/null +++ b/blog/tags/Author-Asumu-Takikawa.html @@ -0,0 +1,202 @@ + + + + + + Posts tagged 'Author: Asumu Takikawa' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Asumu Takikawa

+ +
+
+

Tutorial: Racket FFI, part 3

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +here +and part 2 +here.

+ +

In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.

+ +
+
+
+

Tutorial: Racket FFI, Part 2

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 2 of my tutorial on using the Racket FFI. If you haven’t read +part 1 yet, you can find it +here. +Update: part 3 is also now available +here.

+ +

Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.

+ +
+
+
+

Tutorial: Using Racket’s FFI

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

Update: this post is now part of a series. Part 2 is +here +and part 3 is +here.

+ +

I’ve seen several people ask for a tutorial on Racket’s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +documentation +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.

+ +

With that in mind, this blog post will provide a step-by-step tutorial +for Racket’s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.

+ +
+
+
+

Gradual Typing Across the Spectrum

+

+ :: gradual typing, PI meeting

+

By: Asumu Takikawa

+
+ +
+

Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. — Matthias Felleisen

+ +

Yesterday we hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found here). In case you aren’t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.

+ +

Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing—the ideas that cross language and platform barriers.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Ben-Chung.html b/blog/tags/Author-Ben-Chung.html new file mode 100644 index 00000000..61e200a1 --- /dev/null +++ b/blog/tags/Author-Ben-Chung.html @@ -0,0 +1,159 @@ + + + + + + Posts tagged 'Author: Ben Chung' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Ben Chung

+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+ +
+
+

Fall 2016 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell

+
+ +

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

+ +
    +
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. +
  2. they represent a significant increase in the reasoning power of types over programs.
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Ben-Greenman-2.html b/blog/tags/Author-Ben-Greenman-2.html new file mode 100644 index 00000000..873a68a1 --- /dev/null +++ b/blog/tags/Author-Ben-Greenman-2.html @@ -0,0 +1,265 @@ + + + + + + Posts tagged 'Author: Ben Greenman' (page 2) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Ben Greenman

+ + + + + + +
+
+

Gradual Typing Across the Spectrum, part II

+

+ :: gradual typing, PI meeting

+

By: Ben Greenman

+
+ +

Last week, Northeastern hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:

+ +

http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html

+ +

A common thread among the talks was the question: how to convert a research idea into a tool for software developers?

+ +
+ +
+
+

Continuations

+

+ :: history

+

By: Ben Greenman

+
+ +

From the PRL archives:

+ +
+

It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. — Continuations by Alan Nall, Indiana University, 1983

+ +
+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Ben-Greenman-3.html b/blog/tags/Author-Ben-Greenman-3.html new file mode 100644 index 00000000..2172b306 --- /dev/null +++ b/blog/tags/Author-Ben-Greenman-3.html @@ -0,0 +1,262 @@ + + + + + + Posts tagged 'Author: Ben Greenman' (page 3) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Ben Greenman

+ + + + + + + + + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Ben-Greenman-4.html b/blog/tags/Author-Ben-Greenman-4.html new file mode 100644 index 00000000..09e6973f --- /dev/null +++ b/blog/tags/Author-Ben-Greenman-4.html @@ -0,0 +1,138 @@ + + + + + + Posts tagged 'Author: Ben Greenman' (page 4) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Ben Greenman

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Ben-Greenman.html b/blog/tags/Author-Ben-Greenman.html new file mode 100644 index 00000000..ac7070b1 --- /dev/null +++ b/blog/tags/Author-Ben-Greenman.html @@ -0,0 +1,259 @@ + + + + + + Posts tagged 'Author: Ben Greenman' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Ben Greenman

+ + + + + + + + + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Daniel-Patterson.html b/blog/tags/Author-Daniel-Patterson.html new file mode 100644 index 00000000..c6baf524 --- /dev/null +++ b/blog/tags/Author-Daniel-Patterson.html @@ -0,0 +1,160 @@ + + + + + + Posts tagged 'Author: Daniel Patterson' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Daniel Patterson

+ + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Dustin-Jamner.html b/blog/tags/Author-Dustin-Jamner.html new file mode 100644 index 00000000..f0c9141c --- /dev/null +++ b/blog/tags/Author-Dustin-Jamner.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'Author: Dustin Jamner' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Dustin Jamner

+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Gabriel-Scherer-2.html b/blog/tags/Author-Gabriel-Scherer-2.html new file mode 100644 index 00000000..d4cf8dcd --- /dev/null +++ b/blog/tags/Author-Gabriel-Scherer-2.html @@ -0,0 +1,136 @@ + + + + + + Posts tagged 'Author: Gabriel Scherer' (page 2) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Gabriel Scherer

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Gabriel-Scherer.html b/blog/tags/Author-Gabriel-Scherer.html new file mode 100644 index 00000000..b6a2c547 --- /dev/null +++ b/blog/tags/Author-Gabriel-Scherer.html @@ -0,0 +1,289 @@ + + + + + + Posts tagged 'Author: Gabriel Scherer' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Gabriel Scherer

+ +
+
+

Reviews and author responses: we should stop asking for 500-word responses

+

+ ::

+

By: Gabriel Scherer

+
+ +

This year I reviewed many ICFP submissions, and got to be on the receiving end of equally many author responses (also sometimes called, somewhat combatively, rebuttals). I found that there was a large difference between the official written advice on author responses and what I, as a reviewer reading the responses, found effective. In particular, I now believe that limiting yourself to 500 words should strongly be avoided — we should even stop giving that advice.

+ +
+
+
+

Syntactic parametricity strikes again

+

+ ::

+

By: Gabriel Scherer, Li-Yao Xia

+
+ +

In this blog post, reporting on a collaboration with Li-Yao Xia, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely “syntactic” way, by looking at the structure of canonical derivations. More precisely, I prove that \( +\newcommand{\List}[1]{\mathsf{List}~#1} +\newcommand{\Fin}[1]{\mathsf{Fin}~#1} +\newcommand{\Nat}[1]{\mathbb{N}} +\newcommand{\rule}[2]{\frac{\displaystyle \array{#1}}{\displaystyle #2}} +\newcommand{\judge}[2]{{#1} \vdash {#2}} +\newcommand{\emptyrule}[1]{\begin{array}{c}\\[-1em] #1 \end{array}} + ∀α. \List α → \List \alpha +\) is isomorphic to \( + Π(n:\Nat{}). \List{(\Fin{n})} +\) where \(\Fin{n}\) is the type of integers smaller than \(n\), corresponding to the set \(\{0, 1, \dots, n-1\}\).

+ +
+
+
+

PRL at SNAPL’17

+

+ ::

+

By: Gabriel Scherer

+
+ +

PRL recently produced three papers for the SNAPL conference.

+ + + +
+
+
+

Bullets are good for your Coq proofs

+

+ :: coq

+

By: Gabriel Scherer

+
+ +

I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.

+ +

Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.

+ +
+
+
+

Measuring the submission/review balance

+

+ ::

+

By: Gabriel Scherer

+
+ +

How do researchers know whether they are doing “enough” or “too many” reviews? A measurable goal is to be review-neutral: to have demanded, through our submissions, as many reviews as we have produced as reviewers.

+ +
+
+
+

SRC-submissions

+

+ ::

+

By: Gabriel Scherer

+
+ +

Max New, Daniel Patterson and Ben Greenman recently wrote three two-page abstracts on what they are working on right now. Come have a look — and any feedback is welcome!

+ +
+
+
+

Emacs daemon for fast editor startup

+

+ :: System Administration, Emacs

+

By: Gabriel Scherer

+
+ +

In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs’ bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors — on each git commit for example, or when I use a Firefox extension to edit a textarea content in a proper editor.

+ +

In this blog post, I describe how to use emacsclient to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.

+ +
+ + +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Jan-Vitek.html b/blog/tags/Author-Jan-Vitek.html new file mode 100644 index 00000000..b5d979e7 --- /dev/null +++ b/blog/tags/Author-Jan-Vitek.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'Author: Jan Vitek' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Jan Vitek

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Jonathan-Schuster.html b/blog/tags/Author-Jonathan-Schuster.html new file mode 100644 index 00000000..e974ed53 --- /dev/null +++ b/blog/tags/Author-Jonathan-Schuster.html @@ -0,0 +1,125 @@ + + + + + + Posts tagged 'Author: Jonathan Schuster' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Jonathan Schuster

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Julia-Belyakova.html b/blog/tags/Author-Julia-Belyakova.html new file mode 100644 index 00000000..3dc81c3a --- /dev/null +++ b/blog/tags/Author-Julia-Belyakova.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'Author: Julia Belyakova' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Julia Belyakova

+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Justin-Slepak.html b/blog/tags/Author-Justin-Slepak.html new file mode 100644 index 00000000..c90c4e67 --- /dev/null +++ b/blog/tags/Author-Justin-Slepak.html @@ -0,0 +1,140 @@ + + + + + + Posts tagged 'Author: Justin Slepak' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Justin Slepak

+ +
+
+

Rank Polymorphism

+

+ :: array language

+

By: Justin Slepak

+
+ +

Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let’s just call it lerp) for scalars:

+ +
(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))
+ +

The number marks on each argument indicate the expected “rank” of the argument: how many dimensions it should have. In this case, each one is marked 0, indicating a scalar (i.e., 0-dimensional) argument. The function is usable as-is for

+ +
    +
  • +

    α-blending two RGB pixels

    • +
  • +

    dimming or brightening an image

    • +
  • +

    fade transition between video scenes

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Kevin-Clancy.html b/blog/tags/Author-Kevin-Clancy.html new file mode 100644 index 00000000..039e0996 --- /dev/null +++ b/blog/tags/Author-Kevin-Clancy.html @@ -0,0 +1,140 @@ + + + + + + Posts tagged 'Author: Kevin Clancy' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Kevin Clancy

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Leif-Andersen.html b/blog/tags/Author-Leif-Andersen.html new file mode 100644 index 00000000..30e18c1b --- /dev/null +++ b/blog/tags/Author-Leif-Andersen.html @@ -0,0 +1,195 @@ + + + + + + Posts tagged 'Author: Leif Andersen' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Leif Andersen

+ +
+
+

Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript)

+

+ :: visr, clojure, clojurescript, interactive syntax

+

By: Leif Andersen

+
+ +

+

+ +

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

+ +
Graphical network embedded in text +

Graphical network embedded in text

+ +

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

+ +
    +
  • version control, such as git, works with interactive-syntax;
    • +
  • programs using interactive-syntax can be written and edited with your favorite text editor or IDE;
    • +
  • cut/copy/paste works with interactive-syntax using your operating system’s native clipboard;
    • +
  • code analysis tools, like diff and refactor, still work with interactive-syntax; and
    • +
  • you can use interactive-syntax in any language or environment that supports language-oriented programming.
+ +

To learn more about interactive-syntax, watch this video or read the accompanying paper.

+ + + +

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

+ +

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

+ +
+ +
+
+

Racket 6.9 and Windows 10 Creators Update

+

+ :: racket, windows, bugs

+

By: Leif Andersen

+
+ +

Racket 6.9 was released in April and it has been smooth sailing for many people. However, some people using the Windows 10 Creators Update have been experiencing crashes, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.

+ +

The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.

+ +

A VirtualBox image with Racket preinstalled can be downloaded here:

+ + + +

The username and password for this machine are both racket.

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Li-Yao-Xia.html b/blog/tags/Author-Li-Yao-Xia.html new file mode 100644 index 00000000..80c4a2cb --- /dev/null +++ b/blog/tags/Author-Li-Yao-Xia.html @@ -0,0 +1,138 @@ + + + + + + Posts tagged 'Author: Li-Yao Xia' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Li-Yao Xia

+ +
+
+

Syntactic parametricity strikes again

+

+ ::

+

By: Gabriel Scherer, Li-Yao Xia

+
+ +

In this blog post, reporting on a collaboration with Li-Yao Xia, I will show an example of how some results that we traditionally think of as arising from free theorems / parametricity can be established in a purely “syntactic” way, by looking at the structure of canonical derivations. More precisely, I prove that \( +\newcommand{\List}[1]{\mathsf{List}~#1} +\newcommand{\Fin}[1]{\mathsf{Fin}~#1} +\newcommand{\Nat}[1]{\mathbb{N}} +\newcommand{\rule}[2]{\frac{\displaystyle \array{#1}}{\displaystyle #2}} +\newcommand{\judge}[2]{{#1} \vdash {#2}} +\newcommand{\emptyrule}[1]{\begin{array}{c}\\[-1em] #1 \end{array}} + ∀α. \List α → \List \alpha +\) is isomorphic to \( + Π(n:\Nat{}). \List{(\Fin{n})} +\) where \(\Fin{n}\) is the type of integers smaller than \(n\), corresponding to the set \(\{0, 1, \dots, n-1\}\).

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Matt-Kolosick.html b/blog/tags/Author-Matt-Kolosick.html new file mode 100644 index 00000000..c7690fbc --- /dev/null +++ b/blog/tags/Author-Matt-Kolosick.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'Author: Matt Kolosick' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Matt Kolosick

+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Max-New.html b/blog/tags/Author-Max-New.html new file mode 100644 index 00000000..aa38801b --- /dev/null +++ b/blog/tags/Author-Max-New.html @@ -0,0 +1,174 @@ + + + + + + Posts tagged 'Author: Max New' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Max New

+ +
+
+

Final Algebra Semantics is Observational Equivalence

+

+ :: category theory, math, final encoding, observational equivalence

+

By: Max New

+
+ +

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

+ +

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

+ +
+
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ +
+ +
+
+

Understanding Constructive Galois Connections

+

+ :: icfp, galois connection, adjunction, category theory, math

+

By: Max New

+
+ +

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Milo-Davis.html b/blog/tags/Author-Milo-Davis.html new file mode 100644 index 00000000..08b14716 --- /dev/null +++ b/blog/tags/Author-Milo-Davis.html @@ -0,0 +1,160 @@ + + + + + + Posts tagged 'Author: Milo Davis' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Milo Davis

+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+
+
+

Fall 2016 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell

+
+ +

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

+ +
    +
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. +
  2. they represent a significant increase in the reasoning power of types over programs.
+ +
+
+
+

Beta Reduction (Part 1)

+

+ :: lambda, calculus, beta, reduction, semantics

+

By: Milo Davis

+
+ +

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Ming-Ho-Yee.html b/blog/tags/Author-Ming-Ho-Yee.html new file mode 100644 index 00000000..f1d00b82 --- /dev/null +++ b/blog/tags/Author-Ming-Ho-Yee.html @@ -0,0 +1,232 @@ + + + + + + Posts tagged 'Author: Ming-Ho Yee' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Ming-Ho Yee

+ +
+
+

Four Kinds of Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the first and second parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.

+ +

In this third and final part of my blog series, I’d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.

+ +

This is a “bonus” blog post, because I’m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.

+ +
+
+
+

Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the previous post of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: is R lexically or dynamically scoped?

+ +
+ +
+
+

On-Stack Replacement

+

+ ::

+

By: Ming-Ho Yee

+
+ +

Last semester, I took a course where the final project was to write a survey paper on “a topic in the intersection between computer systems and your area.” So I wrote about on-stack replacement.

+ +
+
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+
+
+

Report: PLISS 2017

+

+ :: pliss, event

+

By: Ming-Ho Yee

+
+ +

Two weeks ago, I attended the first Programming Language Implementation Summer School, held in beautiful Bertinoro, Italy.

+ +

The goal of PLISS was “to prepare early graduate students and advanced undergraduates for research in the field,” and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.

+ +
+ +
+
+

Fall 2016 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell

+
+ +

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

+ +
    +
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. +
  2. they represent a significant increase in the reasoning power of types over programs.
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Olek-Gierczak.html b/blog/tags/Author-Olek-Gierczak.html new file mode 100644 index 00000000..8af01870 --- /dev/null +++ b/blog/tags/Author-Olek-Gierczak.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'Author: Olek Gierczak' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Olek Gierczak

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git "a/blog/tags/Author-Olivier-Flu\314\210ckiger.html" "b/blog/tags/Author-Olivier-Flu\314\210ckiger.html" new file mode 100644 index 00000000..d0010464 --- /dev/null +++ "b/blog/tags/Author-Olivier-Flu\314\210ckiger.html" @@ -0,0 +1,125 @@ + + + + + + Posts tagged 'Author: Olivier Flückiger' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Olivier Flückiger

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Rob-Kleffner.html b/blog/tags/Author-Rob-Kleffner.html new file mode 100644 index 00000000..ffb6f55f --- /dev/null +++ b/blog/tags/Author-Rob-Kleffner.html @@ -0,0 +1,127 @@ + + + + + + Posts tagged 'Author: Rob Kleffner' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Rob Kleffner

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Sam-Caldwell.html b/blog/tags/Author-Sam-Caldwell.html new file mode 100644 index 00000000..e344db00 --- /dev/null +++ b/blog/tags/Author-Sam-Caldwell.html @@ -0,0 +1,179 @@ + + + + + + Posts tagged 'Author: Sam Caldwell' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Sam Caldwell

+ +
+
+

Defining Local Bindings in Turnstile Languages

+

+ :: turnstile, tutorial, language, dsl

+

By: Sam Caldwell

+
+ +

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

+ +

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

+ +

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

+ +

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

+ +
+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+
+
+

Fall 2016 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell

+
+ +

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

+ +
    +
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. +
  2. they represent a significant increase in the reasoning power of types over programs.
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Stephen-Chang.html b/blog/tags/Author-Stephen-Chang.html new file mode 100644 index 00000000..05742561 --- /dev/null +++ b/blog/tags/Author-Stephen-Chang.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'Author: Stephen Chang' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Stephen Chang

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Tony-Garnock-Jones.html b/blog/tags/Author-Tony-Garnock-Jones.html new file mode 100644 index 00000000..ac216483 --- /dev/null +++ b/blog/tags/Author-Tony-Garnock-Jones.html @@ -0,0 +1,150 @@ + + + + + + Posts tagged 'Author: Tony Garnock-Jones' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Tony Garnock-Jones

+ + + +
+
+

History of Actors

+

+ :: history

+

By: Tony Garnock-Jones

+
+ +

Christos Dimoulas is currently teaching a “History of Programming Languages” class at Harvard. The class is, as Christos writes, “definitely not about this”; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.

+ +

On Monday, I presented “the History of Actors” for the class. I’ve made the written-out talk notes and an annotated bibliography available here.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-William-J-Bowman.html b/blog/tags/Author-William-J-Bowman.html new file mode 100644 index 00000000..4af0424d --- /dev/null +++ b/blog/tags/Author-William-J-Bowman.html @@ -0,0 +1,167 @@ + + + + + + Posts tagged 'Author: William J. Bowman' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: William J. Bowman

+ + + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Author-Zeina-Migeed.html b/blog/tags/Author-Zeina-Migeed.html new file mode 100644 index 00000000..eeeff11f --- /dev/null +++ b/blog/tags/Author-Zeina-Migeed.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'Author: Zeina Migeed' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Author: Zeina Migeed

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/CRDTs.html b/blog/tags/CRDTs.html new file mode 100644 index 00000000..9e32f284 --- /dev/null +++ b/blog/tags/CRDTs.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'CRDTs' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged CRDTs

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Emacs.html b/blog/tags/Emacs.html new file mode 100644 index 00000000..3a665262 --- /dev/null +++ b/blog/tags/Emacs.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'Emacs' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Emacs

+ +
+
+

Emacs daemon for fast editor startup

+

+ :: System Administration, Emacs

+

By: Gabriel Scherer

+
+ +

In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs’ bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors — on each git commit for example, or when I use a Firefox extension to edit a textarea content in a proper editor.

+ +

In this blog post, I describe how to use emacsclient to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/FFI.html b/blog/tags/FFI.html new file mode 100644 index 00000000..33c4d6af --- /dev/null +++ b/blog/tags/FFI.html @@ -0,0 +1,184 @@ + + + + + + Posts tagged 'FFI' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged FFI

+ +
+
+

Tutorial: Racket FFI, part 3

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +here +and part 2 +here.

+ +

In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.

+ +
+
+
+

Tutorial: Racket FFI, Part 2

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 2 of my tutorial on using the Racket FFI. If you haven’t read +part 1 yet, you can find it +here. +Update: part 3 is also now available +here.

+ +

Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.

+ +
+
+
+

Tutorial: Using Racket’s FFI

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

Update: this post is now part of a series. Part 2 is +here +and part 3 is +here.

+ +

I’ve seen several people ask for a tutorial on Racket’s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +documentation +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.

+ +

With that in mind, this blog post will provide a step-by-step tutorial +for Racket’s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Gradual-Typing.html b/blog/tags/Gradual-Typing.html new file mode 100644 index 00000000..e6db579e --- /dev/null +++ b/blog/tags/Gradual-Typing.html @@ -0,0 +1,127 @@ + + + + + + Posts tagged 'Gradual Typing' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Gradual Typing

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/HOPL-2.html b/blog/tags/HOPL-2.html new file mode 100644 index 00000000..75659558 --- /dev/null +++ b/blog/tags/HOPL-2.html @@ -0,0 +1,148 @@ + + + + + + Posts tagged 'HOPL' (page 2) + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged HOPL

+ + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/HOPL.html b/blog/tags/HOPL.html new file mode 100644 index 00000000..3b367184 --- /dev/null +++ b/blog/tags/HOPL.html @@ -0,0 +1,244 @@ + + + + + + Posts tagged 'HOPL' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged HOPL

+ + + + + + + + + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/ICFP.html b/blog/tags/ICFP.html new file mode 100644 index 00000000..aba9f1a9 --- /dev/null +++ b/blog/tags/ICFP.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'ICFP' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged ICFP

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/NEPLS.html b/blog/tags/NEPLS.html new file mode 100644 index 00000000..e41affca --- /dev/null +++ b/blog/tags/NEPLS.html @@ -0,0 +1,141 @@ + + + + + + Posts tagged 'NEPLS' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged NEPLS

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/PI-meeting.html b/blog/tags/PI-meeting.html new file mode 100644 index 00000000..6e8371c9 --- /dev/null +++ b/blog/tags/PI-meeting.html @@ -0,0 +1,150 @@ + + + + + + Posts tagged 'PI meeting' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged PI meeting

+ +
+
+

Gradual Typing Across the Spectrum, part II

+

+ :: gradual typing, PI meeting

+

By: Ben Greenman

+
+ +

Last week, Northeastern hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:

+ +

http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html

+ +

A common thread among the talks was the question: how to convert a research idea into a tool for software developers?

+ +
+
+
+

Gradual Typing Across the Spectrum

+

+ :: gradual typing, PI meeting

+

By: Asumu Takikawa

+
+ +
+

Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. — Matthias Felleisen

+ +

Yesterday we hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found here). In case you aren’t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.

+ +

Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing—the ideas that cross language and platform barriers.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/PL-Junior.html b/blog/tags/PL-Junior.html new file mode 100644 index 00000000..d6811c67 --- /dev/null +++ b/blog/tags/PL-Junior.html @@ -0,0 +1,147 @@ + + + + + + Posts tagged 'PL Junior' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged PL Junior

+ +
+
+

Spring 2017 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Matt Kolosick, Dustin Jamner, Artem Pelenitsyn, Julia Belyakova, Sam Caldwell

+
+ +

The PL Junior Seminar is for beginning PhD and interested undergrad and masters students to understand the foundations of programming languages research. It serves to fill in background knowledge and get up to speed with different areas of PL research.

+ +

For the spring 2017 instance of PL Junior we chose program synthesis, the sequent calculus, and logic programming as topics we wanted to learn more about. We also did two group paper readings for Luca Cardelli’s Typeful Programming and Alan Kay’s Early History of Smalltalk. At the same time, we changed up the format from the previous semester.

+ +
+
+
+

Fall 2016 PL Junior Retrospective

+

+ :: PL Junior

+

By: Ben Chung, Milo Davis, Ming-Ho Yee, Sam Caldwell

+
+ +

The Programming Language Seminar, Junior (or “PL Junior”), is a seminar for junior students to learn and discuss topics at a pace more suitable to our background. This semester, we decided to study dependent types. We chose this topic because

+ +
    +
  1. working from the TAPL presentation of type systems, dependent types are a step up in difficulty (excepting F-omega-sub), and
    2. +
  2. they represent a significant increase in the reasoning power of types over programs.
+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/PLT-Redex.html b/blog/tags/PLT-Redex.html new file mode 100644 index 00000000..ebf48391 --- /dev/null +++ b/blog/tags/PLT-Redex.html @@ -0,0 +1,143 @@ + + + + + + Posts tagged 'PLT Redex' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged PLT Redex

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Racket.html b/blog/tags/Racket.html new file mode 100644 index 00000000..96dbbf96 --- /dev/null +++ b/blog/tags/Racket.html @@ -0,0 +1,212 @@ + + + + + + Posts tagged 'Racket' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Racket

+ + + +
+
+

Tutorial: Racket FFI, part 3

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +here +and part 2 +here.

+ +

In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.

+ +
+
+
+

Tutorial: Racket FFI, Part 2

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 2 of my tutorial on using the Racket FFI. If you haven’t read +part 1 yet, you can find it +here. +Update: part 3 is also now available +here.

+ +

Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.

+ +
+
+
+

Tutorial: Using Racket’s FFI

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

Update: this post is now part of a series. Part 2 is +here +and part 3 is +here.

+ +

I’ve seen several people ask for a tutorial on Racket’s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +documentation +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.

+ +

With that in mind, this blog post will provide a step-by-step tutorial +for Racket’s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git "a/blog/tags/R\314\214-JIT.html" "b/blog/tags/R\314\214-JIT.html" new file mode 100644 index 00000000..e8431235 --- /dev/null +++ "b/blog/tags/R\314\214-JIT.html" @@ -0,0 +1,125 @@ + + + + + + Posts tagged 'Ř JIT' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Ř JIT

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Scribble.html b/blog/tags/Scribble.html new file mode 100644 index 00000000..6f2bfb54 --- /dev/null +++ b/blog/tags/Scribble.html @@ -0,0 +1,141 @@ + + + + + + Posts tagged 'Scribble' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Scribble

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/System-Administration.html b/blog/tags/System-Administration.html new file mode 100644 index 00000000..9046c151 --- /dev/null +++ b/blog/tags/System-Administration.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'System Administration' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged System Administration

+ +
+
+

Emacs daemon for fast editor startup

+

+ :: System Administration, Emacs

+

By: Gabriel Scherer

+
+ +

In the early days of the famous Emacs/Vim debates, Emacs was often ridiculed for its bulkiness (Eight Megabytes-of-RAM And Constantly Swapping, etc.). The computational power of our computer has grown much faster than Emacs’ bloat: it takes exactly one second to load on my machine. However, our workflows have also changed, and my workflow implies frequently starting new text editors — on each git commit for example, or when I use a Firefox extension to edit a textarea content in a proper editor.

+ +

In this blog post, I describe how to use emacsclient to reuse an existing Emacs process when creating a new editor window, which reduces editor startup times from 1s to 0.150s on my machine.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Takikawa-constant.html b/blog/tags/Takikawa-constant.html new file mode 100644 index 00000000..1ac92f76 --- /dev/null +++ b/blog/tags/Takikawa-constant.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'Takikawa constant' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Takikawa constant

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/Yoneda.html b/blog/tags/Yoneda.html new file mode 100644 index 00000000..d0ff8e8f --- /dev/null +++ b/blog/tags/Yoneda.html @@ -0,0 +1,134 @@ + + + + + + Posts tagged 'Yoneda' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged Yoneda

+ +
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/about.html b/blog/tags/about.html new file mode 100644 index 00000000..6ca4fbd2 --- /dev/null +++ b/blog/tags/about.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'about' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged about

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/adjunction.html b/blog/tags/adjunction.html new file mode 100644 index 00000000..cd554921 --- /dev/null +++ b/blog/tags/adjunction.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'adjunction' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged adjunction

+ +
+
+

Understanding Constructive Galois Connections

+

+ :: icfp, galois connection, adjunction, category theory, math

+

By: Max New

+
+ +

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/announcement.html b/blog/tags/announcement.html new file mode 100644 index 00000000..d3312b86 --- /dev/null +++ b/blog/tags/announcement.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'announcement' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged announcement

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/array-language.html b/blog/tags/array-language.html new file mode 100644 index 00000000..daa865fe --- /dev/null +++ b/blog/tags/array-language.html @@ -0,0 +1,140 @@ + + + + + + Posts tagged 'array language' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged array language

+ +
+
+

Rank Polymorphism

+

+ :: array language

+

By: Justin Slepak

+
+ +

Rank polymorphism gives you code reuse on arguments of different dimensions. Take a linear interpolation function (let’s just call it lerp) for scalars:

+ +
(λ ((lo 0) (hi 0) (α 0)) (+ (* lo (- 1 α)) (* hi α)))
+ +

The number marks on each argument indicate the expected “rank” of the argument: how many dimensions it should have. In this case, each one is marked 0, indicating a scalar (i.e., 0-dimensional) argument. The function is usable as-is for

+ +
    +
  • +

    α-blending two RGB pixels

    • +
  • +

    dimming or brightening an image

    • +
  • +

    fade transition between video scenes

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/benchmarking.html b/blog/tags/benchmarking.html new file mode 100644 index 00000000..8f7ca952 --- /dev/null +++ b/blog/tags/benchmarking.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'benchmarking' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged benchmarking

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/beta.html b/blog/tags/beta.html new file mode 100644 index 00000000..70714ec7 --- /dev/null +++ b/blog/tags/beta.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'beta' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged beta

+ +
+
+

Beta Reduction (Part 1)

+

+ :: lambda, calculus, beta, reduction, semantics

+

By: Milo Davis

+
+ +

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/books.html b/blog/tags/books.html new file mode 100644 index 00000000..51f5991d --- /dev/null +++ b/blog/tags/books.html @@ -0,0 +1,125 @@ + + + + + + Posts tagged 'books' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged books

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/bugs.html b/blog/tags/bugs.html new file mode 100644 index 00000000..2adf7814 --- /dev/null +++ b/blog/tags/bugs.html @@ -0,0 +1,137 @@ + + + + + + Posts tagged 'bugs' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged bugs

+ +
+
+

Racket 6.9 and Windows 10 Creators Update

+

+ :: racket, windows, bugs

+

By: Leif Andersen

+
+ +

Racket 6.9 was released in April and it has been smooth sailing for many people. However, some people using the Windows 10 Creators Update have been experiencing crashes, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.

+ +

The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.

+ +

A VirtualBox image with Racket preinstalled can be downloaded here:

+ + + +

The username and password for this machine are both racket.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/by-Aaron-Weiss.html b/blog/tags/by-Aaron-Weiss.html new file mode 100644 index 00000000..d6c454eb --- /dev/null +++ b/blog/tags/by-Aaron-Weiss.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Alexi-Turcotte.html b/blog/tags/by-Alexi-Turcotte.html new file mode 100644 index 00000000..d404f89e --- /dev/null +++ b/blog/tags/by-Alexi-Turcotte.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Artem-Pelenitsyn.html b/blog/tags/by-Artem-Pelenitsyn.html new file mode 100644 index 00000000..2bb120fb --- /dev/null +++ b/blog/tags/by-Artem-Pelenitsyn.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Asumu-Takikawa.html b/blog/tags/by-Asumu-Takikawa.html new file mode 100644 index 00000000..70e7635d --- /dev/null +++ b/blog/tags/by-Asumu-Takikawa.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Ben-Chung.html b/blog/tags/by-Ben-Chung.html new file mode 100644 index 00000000..f5c5aa3b --- /dev/null +++ b/blog/tags/by-Ben-Chung.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Ben-Greenman-2.html b/blog/tags/by-Ben-Greenman-2.html new file mode 100644 index 00000000..d62c098b --- /dev/null +++ b/blog/tags/by-Ben-Greenman-2.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Ben-Greenman-3.html b/blog/tags/by-Ben-Greenman-3.html new file mode 100644 index 00000000..0891f028 --- /dev/null +++ b/blog/tags/by-Ben-Greenman-3.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Ben-Greenman-4.html b/blog/tags/by-Ben-Greenman-4.html new file mode 100644 index 00000000..180b6c34 --- /dev/null +++ b/blog/tags/by-Ben-Greenman-4.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Ben-Greenman.html b/blog/tags/by-Ben-Greenman.html new file mode 100644 index 00000000..c8392892 --- /dev/null +++ b/blog/tags/by-Ben-Greenman.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Daniel-Patterson.html b/blog/tags/by-Daniel-Patterson.html new file mode 100644 index 00000000..f361e69d --- /dev/null +++ b/blog/tags/by-Daniel-Patterson.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Dustin-Jamner.html b/blog/tags/by-Dustin-Jamner.html new file mode 100644 index 00000000..5ef38208 --- /dev/null +++ b/blog/tags/by-Dustin-Jamner.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Gabriel-Scherer-2.html b/blog/tags/by-Gabriel-Scherer-2.html new file mode 100644 index 00000000..d5d65f1b --- /dev/null +++ b/blog/tags/by-Gabriel-Scherer-2.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Gabriel-Scherer.html b/blog/tags/by-Gabriel-Scherer.html new file mode 100644 index 00000000..70c39b11 --- /dev/null +++ b/blog/tags/by-Gabriel-Scherer.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Jan-Vitek.html b/blog/tags/by-Jan-Vitek.html new file mode 100644 index 00000000..42587371 --- /dev/null +++ b/blog/tags/by-Jan-Vitek.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Jonathan-Schuster.html b/blog/tags/by-Jonathan-Schuster.html new file mode 100644 index 00000000..deeb27e4 --- /dev/null +++ b/blog/tags/by-Jonathan-Schuster.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Julia-Belyakova.html b/blog/tags/by-Julia-Belyakova.html new file mode 100644 index 00000000..9c9ef864 --- /dev/null +++ b/blog/tags/by-Julia-Belyakova.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Justin-Slepak.html b/blog/tags/by-Justin-Slepak.html new file mode 100644 index 00000000..ca00de2c --- /dev/null +++ b/blog/tags/by-Justin-Slepak.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Kevin-Clancy.html b/blog/tags/by-Kevin-Clancy.html new file mode 100644 index 00000000..2e4e4903 --- /dev/null +++ b/blog/tags/by-Kevin-Clancy.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Leif-Andersen.html b/blog/tags/by-Leif-Andersen.html new file mode 100644 index 00000000..cb5d3152 --- /dev/null +++ b/blog/tags/by-Leif-Andersen.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Li-Yao-Xia.html b/blog/tags/by-Li-Yao-Xia.html new file mode 100644 index 00000000..5752eb0a --- /dev/null +++ b/blog/tags/by-Li-Yao-Xia.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Matt-Kolosick.html b/blog/tags/by-Matt-Kolosick.html new file mode 100644 index 00000000..1829ad86 --- /dev/null +++ b/blog/tags/by-Matt-Kolosick.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Max-New.html b/blog/tags/by-Max-New.html new file mode 100644 index 00000000..fd9fb443 --- /dev/null +++ b/blog/tags/by-Max-New.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Milo-Davis.html b/blog/tags/by-Milo-Davis.html new file mode 100644 index 00000000..fb9d0615 --- /dev/null +++ b/blog/tags/by-Milo-Davis.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Ming-Ho-Yee.html b/blog/tags/by-Ming-Ho-Yee.html new file mode 100644 index 00000000..183904fc --- /dev/null +++ b/blog/tags/by-Ming-Ho-Yee.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Olek-Gierczak.html b/blog/tags/by-Olek-Gierczak.html new file mode 100644 index 00000000..361ee6d4 --- /dev/null +++ b/blog/tags/by-Olek-Gierczak.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Rob-Kleffner.html b/blog/tags/by-Rob-Kleffner.html new file mode 100644 index 00000000..9054b28d --- /dev/null +++ b/blog/tags/by-Rob-Kleffner.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Sam-Caldwell.html b/blog/tags/by-Sam-Caldwell.html new file mode 100644 index 00000000..e1035890 --- /dev/null +++ b/blog/tags/by-Sam-Caldwell.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Stephen-Chang.html b/blog/tags/by-Stephen-Chang.html new file mode 100644 index 00000000..bba7a456 --- /dev/null +++ b/blog/tags/by-Stephen-Chang.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Tony-Garnock-Jones.html b/blog/tags/by-Tony-Garnock-Jones.html new file mode 100644 index 00000000..27aedda2 --- /dev/null +++ b/blog/tags/by-Tony-Garnock-Jones.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-William-J-Bowman.html b/blog/tags/by-William-J-Bowman.html new file mode 100644 index 00000000..57f78518 --- /dev/null +++ b/blog/tags/by-William-J-Bowman.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/by-Zeina-Migeed.html b/blog/tags/by-Zeina-Migeed.html new file mode 100644 index 00000000..2bdff68e --- /dev/null +++ b/blog/tags/by-Zeina-Migeed.html @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/blog/tags/calculus.html b/blog/tags/calculus.html new file mode 100644 index 00000000..619dfc7a --- /dev/null +++ b/blog/tags/calculus.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'calculus' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged calculus

+ +
+
+

Beta Reduction (Part 1)

+

+ :: lambda, calculus, beta, reduction, semantics

+

By: Milo Davis

+
+ +

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/castle.html b/blog/tags/castle.html new file mode 100644 index 00000000..dded72c1 --- /dev/null +++ b/blog/tags/castle.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'castle' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged castle

+ +
+
+

PLISS: Learn About PL Implementation in a Castle

+

+ :: event, lectures, castle, language implementation

+

By: Alexi Turcotte

+
+ +

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

+ +

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/category-theory.html b/blog/tags/category-theory.html new file mode 100644 index 00000000..fee1a042 --- /dev/null +++ b/blog/tags/category-theory.html @@ -0,0 +1,174 @@ + + + + + + Posts tagged 'category theory' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged category theory

+ +
+
+

Final Algebra Semantics is Observational Equivalence

+

+ :: category theory, math, final encoding, observational equivalence

+

By: Max New

+
+ +

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

+ +

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

+ +
+
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ +
+ +
+
+

Understanding Constructive Galois Connections

+

+ :: icfp, galois connection, adjunction, category theory, math

+

By: Max New

+
+ +

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/clojure.html b/blog/tags/clojure.html new file mode 100644 index 00000000..4a3218be --- /dev/null +++ b/blog/tags/clojure.html @@ -0,0 +1,151 @@ + + + + + + Posts tagged 'clojure' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged clojure

+ +
+
+

Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript)

+

+ :: visr, clojure, clojurescript, interactive syntax

+

By: Leif Andersen

+
+ +

+

+ +

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

+ +
Graphical network embedded in text +

Graphical network embedded in text

+ +

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

+ +
    +
  • version control, such as git, works with interactive-syntax;
    • +
  • programs using interactive-syntax can be written and edited with your favorite text editor or IDE;
    • +
  • cut/copy/paste works with interactive-syntax using your operating system’s native clipboard;
    • +
  • code analysis tools, like diff and refactor, still work with interactive-syntax; and
    • +
  • you can use interactive-syntax in any language or environment that supports language-oriented programming.
+ +

To learn more about interactive-syntax, watch this video or read the accompanying paper.

+ + + +

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

+ +

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/clojurescript.html b/blog/tags/clojurescript.html new file mode 100644 index 00000000..5bb22290 --- /dev/null +++ b/blog/tags/clojurescript.html @@ -0,0 +1,151 @@ + + + + + + Posts tagged 'clojurescript' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged clojurescript

+ +
+
+

Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript)

+

+ :: visr, clojure, clojurescript, interactive syntax

+

By: Leif Andersen

+
+ +

+

+ +

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

+ +
Graphical network embedded in text +

Graphical network embedded in text

+ +

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

+ +
    +
  • version control, such as git, works with interactive-syntax;
    • +
  • programs using interactive-syntax can be written and edited with your favorite text editor or IDE;
    • +
  • cut/copy/paste works with interactive-syntax using your operating system’s native clipboard;
    • +
  • code analysis tools, like diff and refactor, still work with interactive-syntax; and
    • +
  • you can use interactive-syntax in any language or environment that supports language-oriented programming.
+ +

To learn more about interactive-syntax, watch this video or read the accompanying paper.

+ + + +

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

+ +

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/closure-conversion.html b/blog/tags/closure-conversion.html new file mode 100644 index 00000000..8f3f3c0d --- /dev/null +++ b/blog/tags/closure-conversion.html @@ -0,0 +1,134 @@ + + + + + + Posts tagged 'closure conversion' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged closure conversion

+ +
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/coYoneda.html b/blog/tags/coYoneda.html new file mode 100644 index 00000000..d8a3b1ce --- /dev/null +++ b/blog/tags/coYoneda.html @@ -0,0 +1,134 @@ + + + + + + Posts tagged 'coYoneda' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged coYoneda

+ +
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/compiler-correctness.html b/blog/tags/compiler-correctness.html new file mode 100644 index 00000000..06a53c79 --- /dev/null +++ b/blog/tags/compiler-correctness.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'compiler correctness' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged compiler correctness

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/compilers.html b/blog/tags/compilers.html new file mode 100644 index 00000000..eb71af35 --- /dev/null +++ b/blog/tags/compilers.html @@ -0,0 +1,134 @@ + + + + + + Posts tagged 'compilers' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged compilers

+ +
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/complete-monitoring.html b/blog/tags/complete-monitoring.html new file mode 100644 index 00000000..f08f5ea0 --- /dev/null +++ b/blog/tags/complete-monitoring.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'complete monitoring' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged complete monitoring

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/conference.html b/blog/tags/conference.html new file mode 100644 index 00000000..f9351a49 --- /dev/null +++ b/blog/tags/conference.html @@ -0,0 +1,141 @@ + + + + + + Posts tagged 'conference' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged conference

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/constructions.html b/blog/tags/constructions.html new file mode 100644 index 00000000..c7b71f70 --- /dev/null +++ b/blog/tags/constructions.html @@ -0,0 +1,133 @@ + + + + + + Posts tagged 'constructions' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged constructions

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/coq.html b/blog/tags/coq.html new file mode 100644 index 00000000..caddcb71 --- /dev/null +++ b/blog/tags/coq.html @@ -0,0 +1,143 @@ + + + + + + Posts tagged 'coq' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged coq

+ +
+
+

Bullets are good for your Coq proofs

+

+ :: coq

+

By: Gabriel Scherer

+
+ +

I believe that bullets are one of the most impactful features of recent versions of Coq, among those that non-super-expert users can enjoy. They had a big impact on the maintainability of my proofs. Unfortunately, they are not very well-known, due to the fact that some introductory documents have not been updated to use them.

+ +

Bullets are a very general construction and there are several possible ways to use them; I have iterated through different styles. In this post I will give the general rules, and explain my current usage style.

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/dawn-of-the-digital-era.html b/blog/tags/dawn-of-the-digital-era.html new file mode 100644 index 00000000..484b2b3f --- /dev/null +++ b/blog/tags/dawn-of-the-digital-era.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'dawn of the digital era' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged dawn of the digital era

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/dear-diary.html b/blog/tags/dear-diary.html new file mode 100644 index 00000000..fbf009f0 --- /dev/null +++ b/blog/tags/dear-diary.html @@ -0,0 +1,141 @@ + + + + + + Posts tagged 'dear diary' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged dear diary

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/definitions.html b/blog/tags/definitions.html new file mode 100644 index 00000000..a274e45a --- /dev/null +++ b/blog/tags/definitions.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'definitions' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged definitions

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/dissertation.html b/blog/tags/dissertation.html new file mode 100644 index 00000000..ba8f6e9a --- /dev/null +++ b/blog/tags/dissertation.html @@ -0,0 +1,143 @@ + + + + + + Posts tagged 'dissertation' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged dissertation

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/dsl.html b/blog/tags/dsl.html new file mode 100644 index 00000000..bace940a --- /dev/null +++ b/blog/tags/dsl.html @@ -0,0 +1,134 @@ + + + + + + Posts tagged 'dsl' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged dsl

+ +
+
+

Defining Local Bindings in Turnstile Languages

+

+ :: turnstile, tutorial, language, dsl

+

By: Sam Caldwell

+
+ +

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

+ +

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

+ +

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

+ +

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/dynamic-typing.html b/blog/tags/dynamic-typing.html new file mode 100644 index 00000000..81c9427b --- /dev/null +++ b/blog/tags/dynamic-typing.html @@ -0,0 +1,127 @@ + + + + + + Posts tagged 'dynamic typing' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged dynamic typing

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/event.html b/blog/tags/event.html new file mode 100644 index 00000000..ecabc310 --- /dev/null +++ b/blog/tags/event.html @@ -0,0 +1,171 @@ + + + + + + Posts tagged 'event' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged event

+ +
+
+

PLISS: Learn About PL Implementation in a Castle

+

+ :: event, lectures, castle, language implementation

+

By: Alexi Turcotte

+
+ +

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

+ +

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

+ +
+ +
+
+

Report: PLISS 2017

+

+ :: pliss, event

+

By: Ming-Ho Yee

+
+ +

Two weeks ago, I attended the first Programming Language Implementation Summer School, held in beautiful Bertinoro, Italy.

+ +

The goal of PLISS was “to prepare early graduate students and advanced undergraduates for research in the field,” and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/eventual-consistency.html b/blog/tags/eventual-consistency.html new file mode 100644 index 00000000..17f577af --- /dev/null +++ b/blog/tags/eventual-consistency.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'eventual consistency' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged eventual consistency

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/extended-abstract.html b/blog/tags/extended-abstract.html new file mode 100644 index 00000000..09d75d63 --- /dev/null +++ b/blog/tags/extended-abstract.html @@ -0,0 +1,153 @@ + + + + + + Posts tagged 'extended abstract' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged extended abstract

+ + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/final-encoding.html b/blog/tags/final-encoding.html new file mode 100644 index 00000000..134572ad --- /dev/null +++ b/blog/tags/final-encoding.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'final encoding' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged final encoding

+ +
+
+

Final Algebra Semantics is Observational Equivalence

+

+ :: category theory, math, final encoding, observational equivalence

+

By: Max New

+
+ +

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

+ +

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/galois-connection.html b/blog/tags/galois-connection.html new file mode 100644 index 00000000..0b08dd37 --- /dev/null +++ b/blog/tags/galois-connection.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'galois connection' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged galois connection

+ +
+
+

Understanding Constructive Galois Connections

+

+ :: icfp, galois connection, adjunction, category theory, math

+

By: Max New

+
+ +

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/garbage-collection.html b/blog/tags/garbage-collection.html new file mode 100644 index 00000000..a973f11d --- /dev/null +++ b/blog/tags/garbage-collection.html @@ -0,0 +1,145 @@ + + + + + + Posts tagged 'garbage collection' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged garbage collection

+ +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/ghc.html b/blog/tags/ghc.html new file mode 100644 index 00000000..2e5e9436 --- /dev/null +++ b/blog/tags/ghc.html @@ -0,0 +1,145 @@ + + + + + + Posts tagged 'ghc' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged ghc

+ +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/gradual-typing.html b/blog/tags/gradual-typing.html new file mode 100644 index 00000000..7fc0af58 --- /dev/null +++ b/blog/tags/gradual-typing.html @@ -0,0 +1,213 @@ + + + + + + Posts tagged 'gradual typing' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged gradual typing

+ + + + + +
+
+

Gradual Typing Across the Spectrum, part II

+

+ :: gradual typing, PI meeting

+

By: Ben Greenman

+
+ +

Last week, Northeastern hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant. The meeting was made of 20+ researchers from four institutions, and 12 technical talks. Schedule:

+ +

http://prl.ccs.neu.edu/gtp/pi2017/pi2017.html

+ +

A common thread among the talks was the question: how to convert a research idea into a tool for software developers?

+ +
+ +
+
+

Gradual Typing Across the Spectrum

+

+ :: gradual typing, PI meeting

+

By: Asumu Takikawa

+
+ +
+

Instead of being Pythonistas, Rubyists, or Racketeers we have to be scientists. — Matthias Felleisen

+ +

Yesterday we hosted a PI meeting for the Gradual Typing Across the Spectrum NSF grant, gathering researchers from a number of institutions who work on gradual typing (the meeting program can be found here). In case you aren’t familiar with gradual typing, the idea is to augment dynamically typed languages (think Python or Ruby) with static type annotations (as documentation, for debugging, or for tool support) that are guaranteed to be sound.

+ +

Gradual typing is these days a fairly popular area, but the research can seem fragmentary because of the need to support idiosyncratic language features. One of the points of the meeting was to encourage the cross-pollination of the key scientific ideas of gradual typing—the ideas that cross language and platform barriers.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/haskell.html b/blog/tags/haskell.html new file mode 100644 index 00000000..ba714c5c --- /dev/null +++ b/blog/tags/haskell.html @@ -0,0 +1,145 @@ + + + + + + Posts tagged 'haskell' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged haskell

+ +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/higher-order-contracts.html b/blog/tags/higher-order-contracts.html new file mode 100644 index 00000000..8be68f27 --- /dev/null +++ b/blog/tags/higher-order-contracts.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'higher-order contracts' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged higher-order contracts

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/history.html b/blog/tags/history.html new file mode 100644 index 00000000..eb5be701 --- /dev/null +++ b/blog/tags/history.html @@ -0,0 +1,193 @@ + + + + + + Posts tagged 'history' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged history

+ + + +
+
+

Continuations

+

+ :: history

+

By: Ben Greenman

+
+ +

From the PRL archives:

+ +
+

It was also a concept that grabbed my mind, ran off with it, and only returned it after substantial renovation and expansion. — Continuations by Alan Nall, Indiana University, 1983

+ +
+ +
+
+

History of Actors

+

+ :: history

+

By: Tony Garnock-Jones

+
+ +

Christos Dimoulas is currently teaching a “History of Programming Languages” class at Harvard. The class is, as Christos writes, “definitely not about this”; instead, each meeting is a deep examination of a single, mature research topic, in terms of three to five key papers from the literature.

+ +

On Monday, I presented “the History of Actors” for the class. I’ve made the written-out talk notes and an annotated bibliography available here.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/icfp.html b/blog/tags/icfp.html new file mode 100644 index 00000000..c6b5db5c --- /dev/null +++ b/blog/tags/icfp.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'icfp' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged icfp

+ +
+
+

Understanding Constructive Galois Connections

+

+ :: icfp, galois connection, adjunction, category theory, math

+

By: Max New

+
+ +

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/instrumentation.html b/blog/tags/instrumentation.html new file mode 100644 index 00000000..5eef2c42 --- /dev/null +++ b/blog/tags/instrumentation.html @@ -0,0 +1,145 @@ + + + + + + Posts tagged 'instrumentation' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged instrumentation

+ +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/interactive-syntax.html b/blog/tags/interactive-syntax.html new file mode 100644 index 00000000..c93cf7e5 --- /dev/null +++ b/blog/tags/interactive-syntax.html @@ -0,0 +1,151 @@ + + + + + + Posts tagged 'interactive syntax' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged interactive syntax

+ +
+
+

Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript)

+

+ :: visr, clojure, clojurescript, interactive syntax

+

By: Leif Andersen

+
+ +

+

+ +

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

+ +
Graphical network embedded in text +

Graphical network embedded in text

+ +

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

+ +
    +
  • version control, such as git, works with interactive-syntax;
    • +
  • programs using interactive-syntax can be written and edited with your favorite text editor or IDE;
    • +
  • cut/copy/paste works with interactive-syntax using your operating system’s native clipboard;
    • +
  • code analysis tools, like diff and refactor, still work with interactive-syntax; and
    • +
  • you can use interactive-syntax in any language or environment that supports language-oriented programming.
+ +

To learn more about interactive-syntax, watch this video or read the accompanying paper.

+ + + +

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

+ +

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/java.html b/blog/tags/java.html new file mode 100644 index 00000000..91d5f306 --- /dev/null +++ b/blog/tags/java.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'java' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged java

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/lambda.html b/blog/tags/lambda.html new file mode 100644 index 00000000..372f2975 --- /dev/null +++ b/blog/tags/lambda.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'lambda' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged lambda

+ +
+
+

Beta Reduction (Part 1)

+

+ :: lambda, calculus, beta, reduction, semantics

+

By: Milo Davis

+
+ +

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/lang-extension.html b/blog/tags/lang-extension.html new file mode 100644 index 00000000..435d06ad --- /dev/null +++ b/blog/tags/lang-extension.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'lang-extension' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged lang-extension

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/language-implementation.html b/blog/tags/language-implementation.html new file mode 100644 index 00000000..c4191daf --- /dev/null +++ b/blog/tags/language-implementation.html @@ -0,0 +1,143 @@ + + + + + + Posts tagged 'language implementation' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged language implementation

+ +
+
+

PLISS: Learn About PL Implementation in a Castle

+

+ :: event, lectures, castle, language implementation

+

By: Alexi Turcotte

+
+ +

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

+ +

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/language.html b/blog/tags/language.html new file mode 100644 index 00000000..7f3c950f --- /dev/null +++ b/blog/tags/language.html @@ -0,0 +1,134 @@ + + + + + + Posts tagged 'language' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged language

+ +
+
+

Defining Local Bindings in Turnstile Languages

+

+ :: turnstile, tutorial, language, dsl

+

By: Sam Caldwell

+
+ +

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

+ +

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

+ +

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

+ +

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/latency.html b/blog/tags/latency.html new file mode 100644 index 00000000..d1b8365d --- /dev/null +++ b/blog/tags/latency.html @@ -0,0 +1,145 @@ + + + + + + Posts tagged 'latency' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged latency

+ +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/lectures.html b/blog/tags/lectures.html new file mode 100644 index 00000000..3bbf0840 --- /dev/null +++ b/blog/tags/lectures.html @@ -0,0 +1,143 @@ + + + + + + Posts tagged 'lectures' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged lectures

+ +
+
+

PLISS: Learn About PL Implementation in a Castle

+

+ :: event, lectures, castle, language implementation

+

By: Alexi Turcotte

+
+ +

We love programming languages (PLs), and we should all be in on the ins and outs of implementing them. If you’re interested in learning the tricks of the trade of PL design and implementation, what better opportunity than the second Programming Languages Implementation Summer School (PLISS for short).

+ +

PLISS will be held from May 19th to 24th 2019, and the deadline to express your interest is March 29th, 2019 at 17:00 GMT. More details can be found here.

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/lost-time.html b/blog/tags/lost-time.html new file mode 100644 index 00000000..66dbd150 --- /dev/null +++ b/blog/tags/lost-time.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'lost time' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged lost time

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/math.html b/blog/tags/math.html new file mode 100644 index 00000000..a0293d68 --- /dev/null +++ b/blog/tags/math.html @@ -0,0 +1,162 @@ + + + + + + Posts tagged 'math' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged math

+ +
+
+

Final Algebra Semantics is Observational Equivalence

+

+ :: category theory, math, final encoding, observational equivalence

+

By: Max New

+
+ +

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

+ +

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

+ +
+
+
+

Closure Conversion as CoYoneda

+

+ :: Yoneda, coYoneda, category theory, compilers, closure conversion, math

+

By: Max New

+
+ +

The continuation-passing style transform (cps) and closure conversion (cc) are two techniques widely employed by compilers for functional languages, and have been studied extensively in the compiler correctness literature. Interestingly, typed versions of each can be proven to be equivalence preserving using polymorphic types and parametric reasoning, as shown by my advisor Amal Ahmed and Matthias Blume (cps,cc).

+ +

In fact, there is something like a duality between the two proofs, cps uses a universal type, closure-conversion uses an existential type and the isomorphism proofs use analogous reasoning. It turns out that both are instances of general theorems in category theory: the polymorphic cps isomorphism can be proven using the Yoneda lemma, and the polymorphic closure-conversion isomorphism can be proven using a less well known theorem often called the *co*Yoneda lemma.

+ +

The connection between cps and the Yoneda embedding/lemma is detailed elsewhere in the literature and blogosphere (ncafe, Bartosz), so I’ll focus on closure conversion here. Also, I’ll try to go into some detail in showing how the “usual” version of Yoneda/coYoneda (using the category of sets) relates to the appropriate version for compilers.

+ +

I’ll assume some background knowledge on closure conversion and parametricity below. Fortunately, Matt Might has a nice blog post explaining untyped closure conversion.

+ +
+
+
+

Understanding Constructive Galois Connections

+

+ :: icfp, galois connection, adjunction, category theory, math

+

By: Max New

+
+ +

One of my favorite papers at ICFP 2016 (in lovely Nara, Japan) was Constructive Galois Connections: Taming the Galois Connection Framework for Mechanized Metatheory by David Darais and David Van Horn. The central technical result is quite interesting, but a little intimidating, so I’d like to share a “de-generalization” of the result that I found helpful to understand.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/migratory-typing.html b/blog/tags/migratory-typing.html new file mode 100644 index 00000000..ad66ccc0 --- /dev/null +++ b/blog/tags/migratory-typing.html @@ -0,0 +1,210 @@ + + + + + + Posts tagged 'migratory typing' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged migratory typing

+ + + + + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/monotonicity.html b/blog/tags/monotonicity.html new file mode 100644 index 00000000..923cdd32 --- /dev/null +++ b/blog/tags/monotonicity.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'monotonicity' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged monotonicity

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/observational-equivalence.html b/blog/tags/observational-equivalence.html new file mode 100644 index 00000000..d9af12a5 --- /dev/null +++ b/blog/tags/observational-equivalence.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'observational equivalence' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged observational equivalence

+ +
+
+

Final Algebra Semantics is Observational Equivalence

+

+ :: category theory, math, final encoding, observational equivalence

+

By: Max New

+
+ +

Recently, “final encodings” and “finally tagless style” have become popular techniques for defining embedded languages in functional languages. In a recent discussion in the Northeastern PRL lab, Michael Ballantyne, Ryan Culpepper and I asked “in what category are these actually final objects”? As it turns out our very own Mitch Wand wrote one of the first papers to make exactly this idea precise, so I read it available here and was pleasantly surprised to see that the definition of a final algebra there is essentially equivalent to the definition of observational equivalence.

+ +

In this post, I’ll go over some of the results of that paper and explain the connection to observational equivalence. In the process we’ll learn a bit about categorical logic, and I’ll reformulate some of the category theory in that paper to be a bit more modern in presentation, cleaning some things up in the process.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/ocaml.html b/blog/tags/ocaml.html new file mode 100644 index 00000000..ca574784 --- /dev/null +++ b/blog/tags/ocaml.html @@ -0,0 +1,145 @@ + + + + + + Posts tagged 'ocaml' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged ocaml

+ +
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/offsite.html b/blog/tags/offsite.html new file mode 100644 index 00000000..d5267478 --- /dev/null +++ b/blog/tags/offsite.html @@ -0,0 +1,141 @@ + + + + + + Posts tagged 'offsite' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged offsite

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/package.html b/blog/tags/package.html new file mode 100644 index 00000000..2c18c0af --- /dev/null +++ b/blog/tags/package.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'package' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged package

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/performance.html b/blog/tags/performance.html new file mode 100644 index 00000000..aed6a27b --- /dev/null +++ b/blog/tags/performance.html @@ -0,0 +1,141 @@ + + + + + + Posts tagged 'performance' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged performance

+ + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/pliss.html b/blog/tags/pliss.html new file mode 100644 index 00000000..a4743f06 --- /dev/null +++ b/blog/tags/pliss.html @@ -0,0 +1,130 @@ + + + + + + Posts tagged 'pliss' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged pliss

+ +
+
+

Report: PLISS 2017

+

+ :: pliss, event

+

By: Ming-Ho Yee

+
+ +

Two weeks ago, I attended the first Programming Language Implementation Summer School, held in beautiful Bertinoro, Italy.

+ +

The goal of PLISS was “to prepare early graduate students and advanced undergraduates for research in the field,” and I think it successfully accomplished that. There were many talks in a variety of areas, such as just-in-time compilers, garbage collection, static analysis, and distributed systems. But PLISS was more than just a series of talks: PLISS provided an environment for interacting with other students as well as senior researchers.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/proceedings.html b/blog/tags/proceedings.html new file mode 100644 index 00000000..2d9e12a7 --- /dev/null +++ b/blog/tags/proceedings.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'proceedings' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged proceedings

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/r.html b/blog/tags/r.html new file mode 100644 index 00000000..929f8cad --- /dev/null +++ b/blog/tags/r.html @@ -0,0 +1,145 @@ + + + + + + Posts tagged 'r' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged r

+ +
+
+

Four Kinds of Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the first and second parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.

+ +

In this third and final part of my blog series, I’d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.

+ +

This is a “bonus” blog post, because I’m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.

+ +
+
+
+

Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the previous post of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: is R lexically or dynamically scoped?

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/racket.html b/blog/tags/racket.html new file mode 100644 index 00000000..93166f54 --- /dev/null +++ b/blog/tags/racket.html @@ -0,0 +1,167 @@ + + + + + + Posts tagged 'racket' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged racket

+ +
+
+

Racket 6.9 and Windows 10 Creators Update

+

+ :: racket, windows, bugs

+

By: Leif Andersen

+
+ +

Racket 6.9 was released in April and it has been smooth sailing for many people. However, some people using the Windows 10 Creators Update have been experiencing crashes, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.

+ +

The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.

+ +

A VirtualBox image with Racket preinstalled can be downloaded here:

+ + + +

The username and password for this machine are both racket.

+ +
+
+
+

Measuring GC latencies in Haskell, OCaml, Racket

+

+ :: garbage collection, latency, instrumentation, haskell, ghc, ocaml, racket

+

By: Gabriel Scherer

+
+ +

James Fisher has a blog post on a case where GHC’s runtime system imposed unpleasant latencies on their Haskell program:

+ +
+

Low latency, large working set, and GHC’s garbage collector: pick two of three

+ +

The blog post proposes a very simple, synthetic benchmark that exhibits the issue — basically, latencies incurred by copy time — with latencies of 50ms that are considered excessive. I thought it would be amusing to reproduce the synthetic benchmark in OCaml and Racket, to see how other GCs handle this.

+ +

Without further ado, the main take-away are as follows: the OCaml GC has no issue with large objects in its old generation, as it uses a mark&sweep instead of copying collection, and exhibits less than 3ms worst-case pauses on this benchmark.

+ +

The Racket GC also does not copy the old generation, but its incremental GC is still in infancy (compared to the throughput-oriented settings which works well) so the results are less good. It currently suffer from a “ramp-up” effect that I will describe, that causes large pauses at the beginning of the benchmark (up to 120ms latency), but in its steady state the longest pause are around 22ms.

+ +

Please keep in mind that the original benchmark is designed to exercise a very specific workflow that exercises worst-case behavior for GHC’s garbage collector. This does not mean that GHC’s latencies are bad in general, or that the other tested languages have smaller latencies in general.

+ +

The implementations I use, with a Makefile encapsulating the logic for running and analyzing them, are available in a Gitlab repository:

+ + + +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/reduction.html b/blog/tags/reduction.html new file mode 100644 index 00000000..622fddb1 --- /dev/null +++ b/blog/tags/reduction.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'reduction' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged reduction

+ +
+
+

Beta Reduction (Part 1)

+

+ :: lambda, calculus, beta, reduction, semantics

+

By: Milo Davis

+
+ +

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/scope.html b/blog/tags/scope.html new file mode 100644 index 00000000..7abecd92 --- /dev/null +++ b/blog/tags/scope.html @@ -0,0 +1,160 @@ + + + + + + Posts tagged 'scope' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged scope

+ +
+
+

Four Kinds of Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the first and second parts of this blog series, I defined lexical and dynamic scope, and demonstrated interesting cases of scoping in R.

+ +

In this third and final part of my blog series, I’d like to discuss a paper by the creators of R, where they motivate the need for lexical scoping in a statistical programming language.

+ +

This is a “bonus” blog post, because I’m going to dive into some of the hairier R features to show how four different kinds of scoping can be simulated in R.

+ +
+
+
+

Scoping in R

+

+ :: scope, r

+

By: Ming-Ho Yee

+
+ +

In the previous post of this three-part blog series, we discussed lexical and dynamic scope. Now, in this second part, we can return to the original question: is R lexically or dynamically scoped?

+ +
+ +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/semantics.html b/blog/tags/semantics.html new file mode 100644 index 00000000..40451afc --- /dev/null +++ b/blog/tags/semantics.html @@ -0,0 +1,138 @@ + + + + + + Posts tagged 'semantics' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged semantics

+ + +
+
+

Beta Reduction (Part 1)

+

+ :: lambda, calculus, beta, reduction, semantics

+

By: Milo Davis

+
+ +

The λ-calculus is often introduced by showing how to build a real programming language from it’s simple syntactic forms. In this series of post, I attempt to introduce it as a tool for modeling semantics. So if you’re opening Barendregt for the first time, trying to understand a lecture from a programming languages or functional programming class, or just starting to become involved in PL research, I hope this post will help you understand evaluation by substitution (β-reduction).

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/statistics.html b/blog/tags/statistics.html new file mode 100644 index 00000000..aa777a28 --- /dev/null +++ b/blog/tags/statistics.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'statistics' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged statistics

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/transient.html b/blog/tags/transient.html new file mode 100644 index 00000000..ae73490c --- /dev/null +++ b/blog/tags/transient.html @@ -0,0 +1,167 @@ + + + + + + Posts tagged 'transient' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged transient

+ + + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/turnstile.html b/blog/tags/turnstile.html new file mode 100644 index 00000000..bcf21b96 --- /dev/null +++ b/blog/tags/turnstile.html @@ -0,0 +1,147 @@ + + + + + + Posts tagged 'turnstile' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged turnstile

+ + +
+
+

Defining Local Bindings in Turnstile Languages

+

+ :: turnstile, tutorial, language, dsl

+

By: Sam Caldwell

+
+ +

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

+ +

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

+ +

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

+ +

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/tutorial.html b/blog/tags/tutorial.html new file mode 100644 index 00000000..48be7bc2 --- /dev/null +++ b/blog/tags/tutorial.html @@ -0,0 +1,280 @@ + + + + + + Posts tagged 'tutorial' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged tutorial

+ + +
+
+

Defining Local Bindings in Turnstile Languages

+

+ :: turnstile, tutorial, language, dsl

+

By: Sam Caldwell

+
+ +

In Racket, programmers can create powerful abstractions by bundling together a family of values, functions, and syntax extensions in the form of a new language. These languages, however, are typically untyped. Turnstile is a new Racket {library,language} for creating typed languages by integrating type checking with Racket’s existing tools for describing languages. The technique is described by fellow PRL’ers in the paper Type Systems as Macros.

+ +

Racket encourages language developers to take full advantage of linguistic reuse by defining new language forms in terms of existing constructs. Unsurprisingly, language extensions often retain some of the Racket-y flavor from the underlying constructs. Implementors save time and energy while users of the language benefit from the familiarity they already have with the Racket ecosystem.

+ +

Unfortunately, Turnstile does not lend itself to expressing one of Racket’s most ubiquitous idioms: naming local bindings with define. Early experience reports from Turnstile, including my own, suggest that language implementors very much desire to include define-like binding forms in their languages.

+ +

This blog post provides a brief overview of what Turnstile is and how it works, an introduction to defining typed language forms, and how to equip these languages with a define binding form.

+ +
+ + + + + +
+
+

Tutorial: Racket FFI, part 3

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 3 of my tutorial for using the Racket FFI. You can find part 1 +here +and part 2 +here.

+ +

In this post, we will experiment with some low-level operations with pointers, +union types, and custom C types. The main takeaway will be the custom C types, +which let you define abstractions that hide the details of the C representation +when manipulating data in Racket.

+ +
+
+
+

Tutorial: Racket FFI, Part 2

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

This is part 2 of my tutorial on using the Racket FFI. If you haven’t read +part 1 yet, you can find it +here. +Update: part 3 is also now available +here.

+ +

Part 2 will continue with more Cairo examples. In this installment, I plan to +go over some more advanced FFI hacking such as handling computed argument +values, custom return arguments, and using C structs.

+ +
+
+
+

Tutorial: Using Racket’s FFI

+

+ :: Racket, FFI, tutorial

+

By: Asumu Takikawa

+
+ +

Update: this post is now part of a series. Part 2 is +here +and part 3 is +here.

+ +

I’ve seen several people ask for a tutorial on Racket’s foreign +function interface (FFI), which allows you to dynamically load +C libraries for use in Racket code. While I think the +documentation +for the FFI is quite good, it is a lot of information to process and +the overview examples may be tricky to run for a beginner.

+ +

With that in mind, this blog post will provide a step-by-step tutorial +for Racket’s FFI that requires minimal setup. All that you will need to +follow along is a copy of Racket and ideally a DrRacket window.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/tutorials.html b/blog/tags/tutorials.html new file mode 100644 index 00000000..3a397cc7 --- /dev/null +++ b/blog/tags/tutorials.html @@ -0,0 +1,125 @@ + + + + + + Posts tagged 'tutorials' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged tutorials

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/typed-racket.html b/blog/tags/typed-racket.html new file mode 100644 index 00000000..af083f56 --- /dev/null +++ b/blog/tags/typed-racket.html @@ -0,0 +1,154 @@ + + + + + + Posts tagged 'typed racket' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged typed racket

+ + + + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/types.html b/blog/tags/types.html new file mode 100644 index 00000000..7d218f2b --- /dev/null +++ b/blog/tags/types.html @@ -0,0 +1,128 @@ + + + + + + Posts tagged 'types' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged types

+ + +
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/visr.html b/blog/tags/visr.html new file mode 100644 index 00000000..4bc54dac --- /dev/null +++ b/blog/tags/visr.html @@ -0,0 +1,151 @@ + + + + + + Posts tagged 'visr' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged visr

+ +
+
+

Introducing Visual and Interactive-Syntax realized (VISr) for ClojureScript (and JavaScript)

+

+ :: visr, clojure, clojurescript, interactive syntax

+

By: Leif Andersen

+
+ +

+

+ +

Visual and interactive-syntax is a type of language-oriented programming that allows developers to use, view, and edit portions of a textual program with graphics. Using interactive-syntax provides the benefits of a graphical programming language, while keeping all of the benefits of a purely textual language. For example, the following is an example of a small network embedded in a program:

+ +
Graphical network embedded in text +

Graphical network embedded in text

+ +

Interactive-syntax is backed by human readable code; the visual components exists purely when writing and editing code. This backing means all of the tools involved in software development work with interactive-syntax extensions. For example:

+ +
    +
  • version control, such as git, works with interactive-syntax;
    • +
  • programs using interactive-syntax can be written and edited with your favorite text editor or IDE;
    • +
  • cut/copy/paste works with interactive-syntax using your operating system’s native clipboard;
    • +
  • code analysis tools, like diff and refactor, still work with interactive-syntax; and
    • +
  • you can use interactive-syntax in any language or environment that supports language-oriented programming.
+ +

To learn more about interactive-syntax, watch this video or read the accompanying paper.

+ + + +

VISr (Visual and Interactive-Syntax realized) for ClojureScript is a practical implementation of interactive-syntax in web browsers. The VISr environment is a full-featured IDE that supports interactive-syntax components called VISrs. Additionally, the VISr environment comes with a package manager that supports NPM packages.

+ +

This article is a brief introduction to both the VISr environment and the components that make up a VISrs. It discusses how to insert a VISr into code, how to manipulate a VISr, and how to create a new types of VISr. Future articles will discuss more advanced uses such as integrating NPM packages and using VISrs in other languages.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/blog/tags/windows.html b/blog/tags/windows.html new file mode 100644 index 00000000..47fc7c3d --- /dev/null +++ b/blog/tags/windows.html @@ -0,0 +1,137 @@ + + + + + + Posts tagged 'windows' + + + + + + + + + + + + + + + + + + + + + +
+
+
+ +
+

Inside PRL

+

Reading between the parentheses

+
+ +
+
+
+
+
+
+
+ + +
+ +

Posts tagged windows

+ +
+
+

Racket 6.9 and Windows 10 Creators Update

+

+ :: racket, windows, bugs

+

By: Leif Andersen

+
+ +

Racket 6.9 was released in April and it has been smooth sailing for many people. However, some people using the Windows 10 Creators Update have been experiencing crashes, not just for Racket, but for the whole operating system. This is due to a bug in Windows. We have contacted Microsoft; they have classified the bug as (1) a stack overflow and (2) not a security hazard, and intend to add a fix in a future version of Windows.

+ +

The next version of Racket will include a patch to help avoid triggering the bug. Until then, one work-around is to run Racket in a virtual machine (VM). This blog post is a step-by-step guide on how to install a VM for Racket.

+ +

A VirtualBox image with Racket preinstalled can be downloaded here:

+ + + +

The username and password for this machine are both racket.

+ +
+
+
+
+
+
+

+

+

Blog generated + by Frog, + using Bootstrap.

+

© Copyright Programming Research Laboratory 2015-2016 | made by Catchexception s.r.o. | source on GitHub

+
+
+
+ + + + + + + \ No newline at end of file diff --git a/contact.html b/contact.html new file mode 100644 index 00000000..c4307b76 --- /dev/null +++ b/contact.html @@ -0,0 +1,120 @@ + + + + + + + + + + +Contact - Programming Research Laboratory - Northeastern University + + + + + + + + + + + + +

Contact

+ +

Who's in charge?

+ +

Inside PRL

+ +

PL Seminar and Internal Talks

Steven Holtzen, Luna Phipps-Costin
+ +

PL Seminar, Junior
+Website

+ Liam DeVoe, Sarah Marshall
+ +

ZED-talks

Cameron Moy
+ +

Applications for Internship, PhD, Postdoc

Jan Vitek
+ +

Website

+ +

Webmaster

Liam DeVoe
+ +

Blog Editor

Katie Hough
+ +

Publication Page Maintainer

Andrew Wagner
+ +

People Page Maintainer

Farideh Khalili
+ +

Teaching & Software Pages Maintainer

Farideh Khalili
+ +

Social

+ +

PRL Social

Michelle Thalakottur
+PRL parties, tea time, bingewatching
+ +

Twitter
+@neu_prl

+ John Li
+ +

Coffee Czar

Minsung Cho, Ryan Doenges
+ +

Tea Tzar

Max Bernstein
+ +

Contact Managers

+ +
John Gouwar
+Mailing Lists: prl-all, prl-students
+GitHub Organization: NuPRL
+ +
Jan Vitek
+Mailing List: prl-staff
+ +

How to find us

+

440 Huntington Ave,
+Boston, MA 02115
+

The Khoury College of Computer Sciences +is located in Building WVH +(aka West Village "H" — don't forget the H, +otherwise you won't be able to distinguish +it from WV "A" through WV "G").

+ +

We are diagonally across the street from the Museum of Fine Arts.

+ +

To get to all Khoury offices, take the elevators + opposite the big glassed-in lab on the first floor.

+ +

The administrative offices and mail boxes are located +in Room 202 WVH, to your right as you get off the elevator +on the second floor.

+ +

The PRL lab is spread across rooms 308 and 330. +Room 308 is behind the glass walls with Racket code on them.

+ +

Talks are often in room 366, located to your right as +you get off the elevator on the third floor.

+ +

+

+

© Copyright Programming Research Laboratory 2015-2021 | made by Catchexception s.r.o. | source on GitHub

+top
+ + + + + + +
diff --git a/index.html b/index.html new file mode 100644 index 00000000..2f56e4e7 --- /dev/null +++ b/index.html @@ -0,0 +1,58 @@ + + + + + + + + + + +Home - Programming Research Laboratory - Northeastern University + + + + + + + + + + + + +
Programming Research Laboratory

Programming Research Laboratory

+

Khoury College of Computer Sciences

+

Northeastern University

+

Boston

+ +

We believe that writing computer programs is the fundamental act of computer science, and that programming languages are therefore our fundamental tool.

+

We seek a deeper understanding of this fundamental tool and how it should be used, and we seek to apply this understanding to the program design process, as well as to novel applications.

+

We take a multi-faceted approach to the study of programming languages and programming methodology, incorporating elements of design, mathematics, experimental science, and engineering.

+

We conduct research on all aspects of programming, including:

+
  • the development of small and large programs
  • the design, implementation, and analysis of programming languages
  • programming environment tools
+ +

Our research program is intertwined with our mission to train undergraduates and graduate students. We routinely exploit research results for our undergraduate courses, and we routinely find research challenges in our teaching.

+ +

+

© Copyright Programming Research Laboratory 2015-2021 | made by Catchexception s.r.o. | source on GitHub

+top
+ + + + + + + diff --git a/new-members.html b/new-members.html new file mode 100644 index 00000000..0ec0e81b --- /dev/null +++ b/new-members.html @@ -0,0 +1,93 @@ + + + + + + + + + + +New Members - Programming Research Laboratory - Northeastern University + + + + + + + + + + + + +

New Members

+

Welcome to the PRL! The instructions below will help you get situated in the lab.

+
  1. Get your student ID (a.k.a. Husky Card) from +a pick-up location. +Visit WVH 202 and/or send mail to operations@ccs.neu.edu to register your card to unlock WVH 308, WVH 330, WVH 366, and your private office (if you have one).
  2. Apply for a Khoury account using +these instructions. +This account comes with a ccs.neu.edu email address and +allows you to access various computers around the college. +Make sure you are able to print using the gaugin (in WVH 308) +or renoir (outside WVH 366) printers; see this page +for help, or visit the Khoury Systems Help Desk in WVH 312.
  3. Coordinate with the lab member in charge of +the people page +(see the Contact page) to add your +picture and bio. Contact also maintainers of the lab mailing lists to +add yourself to appropriate lists: +prl-all for everyone, +prl-students for students, +and prl-staff +for faculty, visiting faculty, and postdocs.
  4. Join the lab's channel on matrix, #prl:matrix.org. +Prospective students are welcome to +ask questions on matrix, too!
  5. For grad students and postdocs: join the lab's Discord channel, by contacting the contact manager.
  6. Sign up for the mailing lists for the PL Seminar and the PL Jr. +seminar (links on the Seminars page), +and check the calendar for upcoming talks. +Sign up for the Reading Group mailing list +(send a request with a Google account or contact the reading group maintainer). +You may also wish to subscribe to the +Harvard PL list (calendar) +and +MIT PL list +to be notified about their talks.
  7. If you need a private place to store code, there is both a +Khoury-managed GitHub instance +and a NuPRL organization +on GitHub. To get access to the nuprl +organization, ask the contact manager to add you as a member.
  8. Follow @neu_prl on Twitter.
  9. For grad students: more information about Northeastern's Khoury College of Computer Sciences can be found on +the PhD Hub wiki, a successor of the grad wiki (login required). +Students are also welcome to join Khoury PHDs Slack +(use your northeastern.edu email address) +and check out Khoury GSA website.

  10. Finally, if you need a place to live, this map (compiled by +Northeastern CS grad students circa 2010) may help. Colors +on shaded regions indicate roughly how good the area is for +grad students (green = good, yellow = okay, red = bad), and +you can click on the regions and pins for more information. +Click here to edit the map. + +Note: This map might contain not up-to-date information.

    + + +

    Also check out this map for average rent prices of the surrounding area. This map is regularly updated.

+

+

© Copyright Programming Research Laboratory 2015-2021 | made by Catchexception s.r.o. | source on GitHub

+top
+ + + + + + + diff --git a/people.html b/people.html new file mode 100644 index 00000000..6cf9bb28 --- /dev/null +++ b/people.html @@ -0,0 +1,482 @@ + + + + + + + + + + +People - Programming Research Laboratory - Northeastern University + + + + + + + + + + + + +

Staff

+ +
+
Mitch Wand
+
Mitch Wand +
+Emeritus Professor +
+wand@ccs.neu.edu +
+http://ccs.neu.edu/home/wand
+
Joined Northeastern, 1985
Joined Indiana University, 1973
PhD, MIT, 1973
BS, MIT, 1969
+

Over the years, I have worked on a variety of problems associated with semantics of programming languages. +Here is a selected list, in roughly reverse chronological order: probabilistic programming languages, binding-safe programming, aspect-oriented programming, analysis-based program transformation, compiler correctness proofs, continuations, macros.

+ +
Karl Lieberherr
+
Karl Lieberherr +
+Professor +
+lieber@ccs.neu.edu +
+http://ccs.neu.edu/home/lieber/
+
Joined Northeastern, 1985
Joined Princeton, 1979
Ph.D. and Diplom ETH Zurich, 1968-1977
+

My current research is about using deep reinforcement learning and Monte Carlo Tree Search (a Google/DeepMind algorithm) to synthesize instance-based programs to solve problems formulated in interpreted first-order logic, e.g., algorithmic problems. +My earlier work in programming methodology centers on using (1) context-free grammars for abstract data types and domain-specific languages (the Demeter Method); (2) a succinct and exact definition of a detector for an interesting class of software design problems: The Law of Demeter (Don't Talk to Strangers); it allows for a direct application when writing object-oriented code; (3) aspect-oriented programming. +I am also interested in problem-solving methodology in algorithms and I helped to develop the classical golden-ratio result for Satisfiability.

+ +
Matthias Felleisen
+
Matthias Felleisen +
+Trustee Professor +
+matthias@ccs.neu.edu +
+http://ccs.neu.edu/home/matthias
+
Joined Northeastern, 2001
Joined Rice, 1987
PhD, Indiana University, 1987
Diplom TH Karlsruhe, 1984
MS University of Arizona, 1981
+

I explore all aspects of program design and programming language design. My current research involves work on behavioral software contracts, gradual typing of scripting languages, language interoperability, language extensibility, and module systems. I also engage in educational outreach work. For the past 20 years, I have worked with middle schools, high schools, after-school programs, and college faculty on injecting design ideas into mathematics and computer science courses. Such educational interactions often inspire our research, and many research efforts end up improving my educational work.

+ +
Olin Shivers
+
Olin Shivers +
+Professor +
+olin@ccs.neu.edu +
+http://www.ccs.neu.edu/home/shivers/
+
Joined Northeastern, 2006
Joined Georgia Tech, 1999
Joined MIT, 1993
PhD, Carnegie Mellon University, 1991
BS, Yale University, 1983
+

My principal research interests include the construction of robust, complex software artifacts and the design of tools that assist programmers in this task; the interaction between systems and programming languages, primarily higher-order typed languages; the design and analysis of programming languages; and compilers. Before coming to Northeastern, I was a research scientist at MIT’s Artificial Intelligence Lab, a founder and CTO of the Smartleaf Corporation, and a faculty member at the Georgia Institute of Technology.

+ +
Amal Ahmed
+
Amal Ahmed +
+Professor +
+amal@ccs.neu.edu +
+http://ccs.neu.edu/home/amal
+
Joined Northeastern, 2011
Joined Indiana University, 2009
Joined Toyota Technological Institute, 2006
Joined Harvard University, 2004
PhD Princeton University, 2004
+

I work on problems involving semantics of programming languages, including advanced type systems for programs that manipulate memory, correct and secure compilation, gradual typing, and language interoperability. My prior work has shown how to scale the logical relations proof method to realistic languages. This technique has been used in numerous contexts, e.g., to prove compiler correctness, to verify concurrent code, to establish guarantees provided by type systems for confidentiality or differential privacy. My present focus is on how to build verified compilers that ensure safe linking of code compiled from different programming languages.

+ +
Ben Lerner
+
Ben Lerner +
+Associate Teaching Professor +
+blerner@ccs.neu.edu +
+
+
Joined Northeastern, 2014
PhD, University of Washington, 2011
+

I have worked on problems in web programming semantics, including designing and analyzing extensibility mechanisms for browsers, studying the interactions between extensions and each other or with intended browser behavior. With colleagues at Brown, I have been helping to design and implement a language that focuses on the linguistic support needed for introductory-level pedagogy.

+ +
Jan Vitek
+
Jan Vitek +
+Professor +
+vitekj@me.com +
+http://janvitek.org
+
Joined Northeastern, 2014
Joined Purdue, 1999
PhD, University of Geneva, 1999
MSc, University of Victoria, 1995
+

I work on the design and implementation of programming languages. I led the implementation of the first real-time Java virtual machine to be flight-tested. With Noble and Potter, I proposed what became known as Ownership Types. I tried to understand JavaScript by dynamic analysis and am now looking at supporting scalable data analysis in R.

+ +
Frank Tip
+
Frank Tip +
+Professor +
+tip@acm.org +
+http://www.franktip.org/
+
Joined Northeastern, 2016
Joined Samsung Research America, 2014
Joined University of Waterloo, 2012
Joined IBM T.J. Watson Research Center, 1995
PhD University of Amsterdam, 1995
+

My research is in the areas of Programming Languages and Software Engineering and is focused on the use of program analysis in tools that help increase programmer productivity and software quality. Specific topics that I've worked on in recent years include tools for detecting and localizing bugs, refactoring, test generation, and optimization.

+ +
Arjun Guha
+
Arjun Guha +
+Associate Professor +
+a.guha@northeastern.edu +
+https://ccs.neu.edu/~arjunguha/
+
Joined Northeastern, 2020
Joined University of Massachusetts Amherst, 2013
Postdoc, Cornell University, 2012–2013
PhD, Brown University, 2012
BA, Grinnell College, 2006
+

I have broad interests in programming languages, but usually work on language-based approaches to security, reliability, and performance. I've spent several years thinking about JavaScript (semantics, type checking, static analysis, reactive programming, and more). I also spent several years working on programming languages for software-defined networking. These days, I study problems that arise in cloud computing, system configurations, and robotics, through the lens of programming languages.

+ +
Jon Bell
+
Jon Bell +
+Assistant Professor +
+jon@jonbell.net +
+https://www.jonbell.net/
+
Joined Northeastern, 2020
Joined George Mason University, 2016
PhD, Columbia University, 2016
M.Phil, Columbia University, 2014
MS, Columbia University, 2011
BS, Columbia University, 2010
+

I apply a systems perspective to software engineering challenges, observing the issues that developer face when creating reliable software, and then designing new mechanisms to support developers. My research focuses on improving existing developer-written tests, making them run faster and more reliably while amplifying them to be more comprehensive and also tracking their overall quality. Rather than focus solely on finding a handful of high-value “million dollar bugs” in a small pool of high-assurance software, my research aims to have very broad impacts, helping everyday developers just as much as experts.

+ +
Steven Holtzen
+
Steven Holtzen +
+Assistant Professor +
+s.holtzen@northeastern.edu +
+https://www.khoury.northeastern.edu/home/sholtzen/
+
Joined Northeastern, 2021
PhD, University of California, Los Angeles, 2021
+

My research focuses on programming languages and artificial intelligence. In particular, my goal is to use programming languages and program analysis techniques as a foundation for specifying and reasoning about probabilistic models. Towards this end I am interested in the design, implementation, and applications of probabilistic programming languages; foundations of probabilistic inference and tractable probabilistic modeling; automated reasoning; and probabilistic verification.

+ +
Chris Martens
+
Chris Martens +
+Associate Professor +
+c.martens@northeastern.edu +
+https://www.khoury.northeastern.edu/~cmartens/
+
Joined Northeastern, 2022
PhD, Carnegie Mellon University, 2015
BS, Carnegie Mellon University, 2008
+

My work focuses on connections between logic and computer science, such as (dependent) type systems, logic programming, logical frameworks, and proof assistants. I use the lens of logic to understand fundamental principles of computation, motivated by supporting elegant abstractions for practitioners of creative endeavors such as mathematics, art, and game design.

+ +
Daniel Patterson
+
Daniel Patterson +
+Assistant Teaching Professor +
+dbp@dbpmail.net +
+https://dbp.io
+
Joined Northeastern Faculty, 2022
PhD, Northeastern University, 2022
+

I'm interested in language and type system interoperability — in particular, languages with very different type systems interacting easily and safely.

+ +
Joshua Gancher
+
Joshua Gancher +
+Assistant Professor +
+j.gancher@northeastern.edu +
+https://gancher.dev
+
Joined Northeastern Faculty, 2024
PhD, Cornell University, 2021
B.A, Reed College, 2016
+

I use tools from the programming languages and verification +literature to create systems with formally proven security guarantees. Among +other things, I am interested in language-based approaches for giving +state-of-the-art security guarantees to high-performance, cryptographic +software.

+ +
Ryan Doenges
+
Ryan Doenges +
+Postdoc +
+r.doenges@northeastern.edu +
+http://ryandoeng.es
+
Joined Northeastern, 2023
PhD, Cornell University, 2023
BS, University of Washington, 2017
+

I study computer systems using tools from programming languages and formal verification, especially domain-specific languages (DSLs). My PhD introduced mechanized semantics and certified translation validators for P4, a networking DSL. My postdoctoral research focuses on DSLs for hardware.

+ +

Students

+ +
+
Michael Ballantyne
+
Michael Ballantyne +
+Advisor: Matthias Felleisen +
+michael.ballantyne@gmail.com +
+http://mballantyne.net
+
Joined Northeastern, 2017
+

I find delight in programming languages that allow extension of their notation, syntactic forms, type systems, runtime behaviors, and development environments. My research aims to build the foundations needed to bring these extensibility features into widespread use.

+ +
Olek Gierczak
+
Olek Gierczak +
+Advisor: Amal Ahmed +
+gierczak.o@northeastern.edu +
+
+
Joined Northeastern, 2019
+

I like reasoning about programming languages and compilers, using pencil and paper or proof assistants.

+ +
Cameron Moy
+
Cameron Moy +
+Advisor: Matthias Felleisen +
+camoy@ccs.neu.edu +
+http://camoy.name
+
Joined Northeastern, 2019
+

I enjoy flexible programming languages and elegant software. I'm interested in designing tools that enable everyday developers to build more robust programs.

+ +
Sam Stites
+
Sam Stites +
+Advisor: Steven Holtzen +
+prl@s.ptr.dev +
+
+
Joined Northeastern, 2019
+

I research probabilistic programming languages, specifically designing languages that let users fine-tune the (statistical) inference process.

+ +
Nate Yazdani
+
Nate Yazdani +
+Advisor: Amal Ahmed +
+nyazdani@ccs.neu.edu +
+https://nateyazdani.github.io/
+
Joined Northeastern, 2019
MS, University of Washington, 2019
+

I like types, proofs, and occasionally programs.

+ +
Andrew Wagner
+
Andrew Wagner +
+Advisor: Amal Ahmed +
+ahwagner@ccs.neu.edu +
+
+
Joined Northeastern, 2020
BS, Brown University, 2020
+

I like to design domain-specific languages with strong guarantees, which are usually supported by rich type systems and formal methods. My current research is on language-based security.

+ +
Katherine Hough
+
Katherine Hough +
+Advisor: Jonathan Bell +
+hough.k@northeastern.edu +
+
+
Joined Northeastern, 2020
MS, George Mason University
BS, George Mason University
+

My research focuses on helping developers identify and correct software bugs and vulnerabilities.

+ +
Donald Pinckney
+
Donald Pinckney +
+Advisor: Arjun Guha +
+donald_pinckney@icloud.com +
+https://donaldpinckney.com
+
Joined Northeastern, 2020
MS, UMass Amherst
BS, UC Davis
+

I enjoy working on formalizing semantics of systems so as to uncover surprising behavior, and fix related bugs. Recently I'm working on understanding the semantics of package managers.

+ +
Michelle Thalakottur
+
Michelle Thalakottur +
+Advisor: Amal Ahmed, Frank Tip +
+michelledt@ccs.neu.edu +
+https://michelledaviest.github.io/
+
Joined Northeastern, 2021
+

I like thinking about programming languages and compilers.

+ +
John Gouwar
+
John Gouwar +
+Advisor: Arjun Guha +
+gouwar.j@northeastern.edu +
+https://johngouwar.github.io/
+
Joined Northeastern, 2021
BA, Grinnell College
+

I really enjoy functional programming and would like to develop languages that allow for the use of functional programming techniques in areas that they have not been used before.

+ +
John Li
+
John Li +
+Advisor: Amal Ahmed +
+johnli0135@gmail.com +
+
+
Joined Northeastern, 2021
+

I like logic, semantics, and interactive theorem proving.

+ +
Satyajit Gokhale
+
Satyajit Gokhale +
+Advisor: Frank Tip +
+gokhale.sa@northeastern.edu +
+
+
Joined PRL, 2021
MS, Northeastern University
+

I am interested in program analysis and security. I am currently working on static analysis for JavaScript, and security in PLCs.

+ +
James Perretta
+
James Perretta +
+Advisors: Jonathan Bell and Arjun Guha +
+perretta.j@northeastern.edu +
+
+
Joined Northeastern, 2021
MS, University of Michigan
+

I completed my master's degree at University of Michigan, where I also developed an automated grading system that is used by over 5000 students per semester. My research interests lie at the intersection of PL and Software Engineering, and my current work is focused on mutation testing.

+ +
Yangtian Zi
+
Yangtian Zi +
+Advisor: Arjun Guha +
+ytzi@ccs.neu.edu +
+http://ytzi.org
+
Joined Northeastern, 2021
+

My interests are dynamic languages implementations, Just-in-time compilers, and WebAssembly.

+ +
Gwenyth Lincroft
+
Gwenyth Lincroft +
+Advisor: Jonathan Bell +
+lincroft.g@northeastern.edu +
+
+
Joined Northeastern, 2022
BS, NC State University
+

I am interested in improving tools used by data scientists.

+ +
Francesca Lucchetti
+
Francesca Lucchetti +
+Advisor: Arjun Guha +
+lucchetti.f@northeastern.edu +
+
+
Joined Northeastern, 2022
BA, Vassar College, 2022
+

I’m interested in developing robust, performant Large Language Models for code that can be deployed with limited compute. I like taking apart transformer layers to interpret their inner workings.

+ +
Farideh Khalili
+
Farideh Khalili +
+Advisor: Frank Tip +
+khalili.f@northeastern.edu +
+
+
Joined Northeastern, 2023
+

I'm interested in program analysis and optimization.

+ +
Liam DeVoe
+
Liam DeVoe +
+Advisor: Jonathan Bell +
+devoe.l@northeastern.edu +
+https://tybug.github.io/
+
Joined Northeastern, 2023
B.S. University of Maryland
+

I'm interested in property based testing, fuzzing, and testing of all kinds.

+ +
Shubh Agrawal
+
Shubh Agrawal +
+Advisor: Steven Holtzen +
+agrawal.shub@northeastern.edu +
+
+
Joined Northeastern, 2024
BSE, University of Michigan
+

I'm interested in type systems, logic, dynamics, and abstract semantics for proving interesting properties about interesting programming languages.

+ +
Conrad Zimmerman
+
Conrad Zimmerman +
+Advisor: Amal Ahmed +
+zimmerman.co@northeastern.edu +
+https://conradz.com
+
Joined Northeastern, 2024
BS Brown University, 2024
AS Harrisburg Area Community College, 2020
+

I'm interested in developing approaches that enable both computers and humans to more easily reason about complex applications.

+
Harshit Garg
+
Harshit Garg +
+Advisor: Frank Tip +
+garg.hars@northeastern.edu +
+https://harshitgarg.in
+
Joined Northeastern, 2024
B.E., BITS Pilani
+

I like to work on programming languages, and software testing. I also enjoy thinking about specifications of distributed systems.

+
Vadym Matviichuk
+
Vadym Matviichuk +
+Advisor: Olin Shivers +
+matviichuk.v@northeastern.edu +
+
+
Joined Northeastern, 2024
+

I primarily work on compilers, trying to make programs run fast, and I am also interested in adjacent fields like Programming Languages. +Currently working on a compiler that produces parallel GPU code, so developers don't have to use C++ and think about hardware

+
Luis Garcia
+
Luis Garcia +
+Advisor: Chris Martens +
+garcia.lui@northeastern.edu +
+https://himluis.com
+
Joined Northeastern, 2022
+

I study linear logic and its relationship to other formalisms like planning.

+ +

PRL Alumni and Former Members

+
+ +

Former Associates (Research Scientists, Post-Docs, Visiting Faculty)

+
+ +

Former Faculty

+
+

+

© Copyright Programming Research Laboratory 2015-2021 | made by Catchexception s.r.o. | source on GitHub

+top
+ + + + + + +
diff --git a/publications.html b/publications.html new file mode 100644 index 00000000..39b559b5 --- /dev/null +++ b/publications.html @@ -0,0 +1,1788 @@ + + + + + + + + + + +Publications - Programming Research Laboratory - Northeastern University + + + + + + + + + + + + +

Publications

+ +
2024 +
+
How Beginning Programmers and Code LLMs (Mis)read Each Other +[link ] +
+Sydney Nguyen, Hannah McLean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q Feldman +
+ACM Conference on Human Factors in Computing Systems (CHI)
NL2Code-Reasoning And Planning With LLMs For Code Development +[link ] +
+Ye Xing, Jun Huan, Wee Hyong Tok, Cong Shen, Johannes Gehrke, Katherine Lin, Arjun Guha, Omer Tripp, and Murali Krishna Ramanathan +
+Conference on Knowledge Discovery and Data Mining (KDD)
Semantic Matching In GUI Test Reuse +[link ] +
+Farideh Khalili, Leonardo Mariani, Ali Mohebbi, Mauro Pezzè, and Valerio Terragni +
+Empirical Software Engineering (ESE)
StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code +[link ] +
+Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Arjun Guha, Molly Q Feldman, and Carolyn Jane Anderson +
+Findings of the Association for Computational Linguistics (ACL Findings)
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation +[link ] +
+Max Schaefer, Sarah Nadi, Aryaz Eghbali, and Frank Tip +
+IEEE Transactions on Software Engineering (TSE)
Crossover in Parametric Fuzzing +[link ] +
+Katherine Hough and Jonathan Bell +
+International Conference on Software Engineering (ICSE)
230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers +[link ] +
+Abdulrahman Alshammari, Paul Ammann, Michael Hilton, and Jonathan Bell +
+International Conference on Software Testing, Verification and Validation (ICST)
Automatically Reproducing Timing-Dependent Flaky-Test Failures +[link ] +
+Shanto Rahman, Aaron Massey, Wing Lam, August Shi, and Jonathan Bell +
+International Conference on Software Testing, Verification and Validation (ICST)
Ahead-Of-Time Compilation For Diverse Samplers Of Constrained Design Spaces +[link ] +
+Abdelrahman Madkour, Ross Mawhorter, Stacy Marsella, Adam M. Smith, and Steven Holtzen +
+International Conference on the Foundations of Digital Games (FDG)
Knuth-Morris-Pratt Illustrated +[link ] +
+Cameron Moy +
+Journal of Functional Programming (JFP)
Trace Contracts +[link ] +
+Cameron Moy and Matthias Felleisen +
+Journal of Functional Programming (JFP)
A Nominal Approach To Probabilistic Separation Logic +[link ] +
+John M. Li, Jon Aytac, Philip Johnson-Freyd, Amal Ahmed, and Steven Holtzen +
+Logic in Computer Science (LICS)
Thirty-Three Years of Mathematicians and Software Engineers: A Case Study of Domain Expertise and Participation in Proof Assistant Ecosystems +[link ] +
+Gwenyth Lincroft, Minsung Cho, Katherine Hough, Mahsa Bazzaz, and Jonathan Bell +
+Mining Software Repositories (MSR)
Realistic Realizability: Specifying ABIs You Can Count On +[link ] +
+Andrew Wagner, Zachary Eisbach, and Amal Ahmed +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Forge: A Tool And Language For Teaching Formal Methods +[link ] +
+Tim Nelson, Ben Greenman, Siddhartha Prasad, Tristan Dyer, Ethan Bove, Qianfan Chen, Charles Cutting, Thomas Del Vecchio, Sidney Levine, Julianne Rudner, Ben Ryjikov, Alexander Varga, Andrew Wagner, Luke West, and Shriram Krishnamurthi +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Gradually Typed Languages Should Be Vigilant! +[link ] +
+Olek Gierczak, Lucy Menon, Christos Dimoulas, and Amal Ahmed +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Effectful Software Contracts +[link ] +
+Cameron Moy, Christos Dimoulas, and Matthias Felleisen +
+Principles of Programming Languages (POPL)
Decidable Subtyping Of Existential Types For Julia +[link ] +
+Julia Belyakova, Benjamin Chung, Ross Tate, and Jan Vitek +
+Programming Language Design and Implementation (PLDI)
RichWasm: Bringing Safe, Fine-Grained, Shared-Memory Interoperability Down To Webassembly +[link ] +
+Michael Fitzgibbons, Zoe Paraskevopoulou, Noble Mushtak, Michelle Thalakottur, Jose Sulaiman Manzur, and Amal Ahmed +
+Programming Language Design and Implementation (PLDI)
Bit Blasting Probabilistic Programs +[link ] +
+Poorva Garg, Steven Holtzen, Guy Van den Broeck, and Todd D. Millstein +
+Programming Language Design and Implementation (PLDI)
Deploying and Evaluating LLMs to Program Service Mobile Robots +[link ] +
+Zichao Hu, Francesca Lucchetti, Claire Schlesinger, Yash Saxena, Anders Freeman, Sadanand Modak, Arjun Guha, and Joydeep Biswas +
+Robotics and Automation Letters
2023 +
+
Probabilistic Logic Programming Semantics For Procedural Content Generation +[link ] +
+Abdelrahman Madkour, Chris Martens, Steven Holtzen, Casper Harteveld, and Stacy Marsella +
+Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE)
Scaling Integer Arithmetic in Probabilistic Programs +[link ] +
+William X. Cao, Poorva Garg, Ryan Tjoa, Steven Holtzen, Todd D. Millstein, and Guy Van den Broeck +
+Conference on Uncertainty in Artificial Intelligence (UAI)
SantaCoder: don't reach for the stars! +[link ] +
+Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra +
+Deep Learning for Code Workshop (DL4C)
Injecting Language Workbench Technology Into Mainstream Languages +[link ] +
+Michael Ballantyne and Matthias Felleisen +
+Eelco Visser Commemorative Symposium (EVCS)
Do Machine Learning Models Produce TypeScript Types That Type Check? +[link ] +
+Ming-Ho Yee and Arjun Guha +
+European Conference on Object-Oriented Programming (ECOOP)
Code Coverage Criteria for Asynchronous Programs +[link ] +
+Mohammad Ganji, Saba Alimadadi, and Frank Tip +
+European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)
MultiPL-E: A Scalable And Polyglot Approach To Benchmarking Neural Code Generation +[link ] +
+Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda +
+IEEE Transactions on Software Engineering (TSE)
Increasing the Responsiveness of Web Applications by Introducing Lazy Loading +[link ] +
+Alexi Turcotte, Satyajit Gokhale, and Frank Tip +
+International Conference on Automated Software Engineering (ASE)
How to Evaluate Blame for Gradual Types, Part 2 +[link ] +
+Lukas Lazarek, Ben Greenman, Matthias Felleisen, and Christos Dimoulas +
+International Conference on Functional Programming (ICFP)
Flexible and Optimal Dependency Management via Max-SMT +[link ] +
+Donald Pinckney, Federico Cassano, Arjun Guha, Jonathan Bell, Massimiliano Culpo, and Todd Gamblin +
+International Conference on Software Engineering (ICSE)
That's a Tough Call: Studying the Challenges of Call Graph Construction for WebAssembly +[link ] +
+Daniel Lehmann, Michelle Thalakottur, Frank Tip, and Michael Pradel +
+International Symposium on Software Testing and Analysis (ISSTA)
Semantic Encapsulation using Linking Types +[link ] +
+Daniel Patterson, Andrew Wagner, and Amal Ahmed +
+International Workshop on Type-Driven Development (TyDE)
Approximating Type Stability In The Julia Jit (Work In Progress) +[link ] +
+Artem Pelenitsyn +
+International Workshop on Virtual Machines and Intermediate Languages (VMIL)
A Large Scale Analysis of Semantic Versioning in NPM +[link ] +
+Donald Pinckney, Federico Cassano, Arjun Guha, and Jonathan Bell +
+Mining Software Repositories (MSR)
How Profilers Can Help Navigate Type Migration +[link ] +
+Ben Greenman, Matthias Felleisen, and Christos Dimoulas +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Reusing Just-in-Time Compiled Code +[link ] +
+Meetesh Kalpesh Mehta, Sebastián Krynski, Hugo Musso Gualandi, Manas Thakur, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Continuing WebAssembly with Effect Handlers +[link ] +
+Luna Phipps-Costin, Andreas Rossberg, Arjun Guha, Daan Leijen, Daniel Hillerström, KC Sivaramakrishnan, Matija Pretnar, and Sam Lindle +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Lilac: A Modal Separation Logic for Conditional Probability +[link ] +
+John M. Li, Amal Ahmed, Steven Holtzen +
+Programming Language Design and Implementation (PLDI)
StarCoder: May the Source be With You! +[link ] +
+Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Randy, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Suriya Gunasekar, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries +
+Transactions on Machine Learning Research (TMLR)
Typed–Untyped Interactions: A Comparative Analysis +[link ] +
+Ben Greenman, Christos Dimoulas, and Matthias Felleisen +
+Transactions on Programming Languages and Systems (TOPLAS)
Faster, Simpler Red-Black Trees +[link ] +
+Cameron Moy +
+Trends in Functional Programming (TFP)
2022 +
+
Adversary Safety By Construction In A Language Of Cryptographic Protocols +[link ] +
+Timothy M. Braje, Alice R. Lee, Andrew Wagner, Benjamin Kaiser, Daniel Park, Martine Kalke, Robert K. Cunningham, and Adam Chlipala +
+Computer Security Foundations Symposium (CSF)
Stubbifier: Debloating Dynamic Server-Side JavaScript Applications +[link ] +
+Alexi Turcotte, Ellen Arteca, Ashish Mishra, Saba Alimadadi, and Frank Tip +
+Empirical Software Engineering
A Retrospective Study Of One Decade Of Artifact Evaluations +[link ] +
+Stefan Winter, Christopher Steven Timperley, Ben Hermann, Jürgen Cito, Jonathan Bell, Michael Hilton, and Dirk Beyer +
+European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)
Learning How to Listen: Automatically Finding Bug Patterns in Event-Driven JavaScript APIs +[link ] +
+Ellen Arteca, Max Schäfer, and Frank Tip +
+IEEE Transactions on Software Engineering (TSE)
Reformulator: Automated Refactoring Of The N+1 Problem In Database-Backed Applications +[link ] +
+Alexi Turcotte, Mark W. Aldrich, and Frank Tip +
+International Conference on Automated Software Engineering (ASE)
Augur: Dynamic Taint Analysis For Asynchronous Javascript +[link ] +
+Mark W. Aldrich, Alexi Turcotte, Matthew Blanco, and Frank Tip +
+International Conference on Automated Software Engineering (ASE)
Analyzing Binding Extent in 3CPS +[link ] +
+Benjamin Quiring, Olin Shivers, and John Reppy +
+International Conference on Functional Programming (ICFP)
DrAsync: Identifying and Visualizing Anti-Patterns in Asynchronous JavaScript +[link ] +
+Alexi Turcotte, Michael D. Shah, Mark W. Aldrich, and Frank Tip +
+International Conference on Software Engineering (ICSE)
Nessie: Automatically Testing JavaScript APIs with Asynchronous Callbacks +[link ] +
+Ellen Arteca, Sebastian Harner, Michael Pradel, and Frank Tip +
+International Conference on Software Engineering (ICSE)
CONFETTI: Amplifying Concolic Guidance for Fuzzers +[link ] +
+James Kukucka, Luís Pina, Paul Ammann, and Jonathan Bell +
+International Conference on Software Engineering (ICSE)
signatr: A Data-Driven Fuzzing Tool For R +[link ] +
+Alexi Turcotte, Pierre Donat-Bouillud, Filip Křikava, and Jan Vitek +
+International Conference on Software Language Engineering (SLE)
On the Use of Mutation Analysis For Evaluating Student Test Suite Quality +[link ] +
+James Perretta, Andrew DeOrio, Arjun Guha, and Jonathan Bell +
+International Symposium on Software Testing and Analysis (ISSTA)
ANF Preserves Dependent Types up to Extensional Equality +[link ] +
+Paulette Koronkevich, Ramon Rakow, Amal Ahmed, and William J. Bowman +
+Journal of Functional Programming (JFP)
Computing Correctly With Inductive Relations +[link ] +
+Zoe Paraskevopoulou, Aaron Eline, and Leonidas Lampropoulos +
+Programming Language Design and Implementation (PLDI)
Deoptless: Speculation with Dispatched On-Stack Replacement and Specialized Continuations +[link ] +
+Olivier Flückiger, Jan Ječmen, Sebastián Krynski, and Jan Vitek +
+Programming Language Design and Implementation (PLDI)
Semantic Soundness for Language Interoperability +[link ] +
+Daniel Patterson, Noble Mushtak, Andrew Wagner, and Amal Ahmed +
+Programming Language Design and Implementation (PLDI)
Integrated Data Science for Secondary Schools: Design and Assessment of a Curriculum +[link ] +
+Emmanuel Schanzer, Nancy Pfenning, Flannery Denny, Sam Dooman, Joe Gibbs Politz, Benjamin S. Lerner, Kathi Fisler, and Shriram Krishnamurthi +
+Technical Symposium on Computer Science Education (SIGCSE)
A Transient Semantics For Typed Racket +[link ] +
+Ben Greenman, Lukas Lazarek, Christos Dimoulas, and Matthias Felleisen +
+The Art, Science, and Engineering of Programming
2021 +
+
A Practical Approach for Dynamic Taint Tracking with Control-flow Relationships +[link ] +
+Katherine Hough and Jonathan Bell +
+ACM Transactions on Software Engineering and Methodology (TOSEM)
First-Class Environments in R +[link ] +
+Aviral Goel and Jan Vitek +
+Dynamic Languages Symposium (DLS)
Accelerating Graph Sampling For Graph Machine Learning Using GPUs +[link ] +
+Abhinav Jangda, Sandeep Polisetty, Arjun Guha, and Marco Serafini +
+European Conference on Computer Systems (EuroSys)
CodeDJ: Reproducible Queries over Large-Scale Software Repositories +[link ] +
+Petr Maj, Konrad Siek, Jan Vitek, and Alexander Kovalenko +
+European Conference on Object-Oriented Programming (ECOOP)
Enabling Additional Parallelism in Asynchronous JavaScript Applications +[link ] +
+Ellen Arteca, Frank Tip, and Max Schäefer +
+European Conference on Object-Oriented Programming (ECOOP)
3CPS: The Design of an Environment-Focussed Intermediate Representation +[link ] +
+Benjamin Quiring, John Reppy, and Olin Shivers +
+Implementation and Application of Functional Languages (IFL)
How to Evaluate Blame for Gradual Types +[link ] +
+Lukas Lazarek, Ben Greenman, Matthias Felleisen, and Christos Dimoulas +
+International Conference on Functional Programming (ICFP)
Compositional Optimizations for Certicoq +[link ] +
+Zoe Paraskevopoulou, John M. Li, and Andrew W. Appel +
+International Conference on Functional Programming (ICFP)
Iterative Program Synthesis For Adaptable Social Navigation +[link ] +
+Jarrett Holtz, Simon Andrews, Arjun Guha, and Joydeep Biswas +
+International Conference on Intelligent Robots and Systems (IROS)
FlakeFlagger: Predicting Flakiness Without Rerunning Tests +[link ] +
+Abdulrahman Alshammari, Christopher Morris, Michael Hilton, and Jonathan Bell +
+International Conference on Software Engineering (ICSE)
Cryptographic Hardness Under Projections For Time-Bounded Kolmogorov Complexity +[link ] +
+Eric Allender, John Gouwar, Shuichi Hirahara, and Caleb Robelle +
+International Symposium on Algorithms and Computation (ISAAC)
Coarsening Optimization for Differentiable Programming +[link ] +
+Xipeng Shen, Guoqiang Zhang, Irene Dea, Samantha Andow, Emilio Arroyo-Fang, Neal Gafter, Johann George, Melissa Grueter, Erik Meijer, Olin Shivers, Steffi Stumpos, Alanna Tempest, Christy Warden, and Shannon Yang +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
What We Eval in the Shadows: A Large-Scale Study of Eval in R Programs +[link ] +
+Aviral Goel, Pierre Donat-Bouillud, Filip Křikava, Christoph M. Kirsch, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Type Stability in Julia: Avoiding Performance Pathologies in JIT Compilation +[link ] +
+Artem Pelenitsyn, Julia Belyakova, Benjamin Chung, Ross Tate, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Promises Are Made to Be Broken: Migrating R to Strict Semantics +[link ] +
+Aviral Goel, Jan Ječmen, Sebastián Krynski, Olivier Flückiger, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Solver-Based Gradual Type Migration +[link ] +
+Luna Phipps-Costin, Carolyn Jane Anderson, Michael Greenberg, and Arjun Guha +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Automatic Migration from Synchronous to Asynchronous Javascript APIs +[link ] +
+Satyajit Gokhale, Alexi Turcotte, and Frank Tip +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Compiling with Continuations, Correctly +[link ] +
+Zoe Paraskevopoulou and Anvay Grover +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Corpse Reviver: Sound And Efficient Gradual Typing Via Contract Verification +[link ] +
+Cameron Moy, Phúc C. Nguyễn, Sam Tobin-Hochstadt, and David Van Horn +
+Principles of Programming Languages (POPL)
Formally Verified Speculation and Deoptimization in a JIT Compiler +[link ] +
+Aurèle Barrière, Olivier Flückiger, Sandrine Blazy, David Pichardie, and Jan Vitek +
+Principles of Programming Languages (POPL)
Proof Repair Across Type Equivalences +[link ] +
+Talia Ringer, RanDair Porter, Nathaniel Yazdani, John Leo, and Dan Grossman +
+Programming Language Design and Implementation (PLDI)
Evolving A K-12 Curriculum For Integrating Computer Science Into Mathematics +[link ] +
+Kathi Fisler, Emmanuel Schanzer, Steve Weimar, Annie Fetter, K. Ann Renninger, Shriram Krishnamurthi, Joe Gibbs Politz, Benjamin Lerner, Jennifer Poole, and Christine Koerner +
+Technical Symposium on Computer Science Education (SIGCSE)
2020 +
+
Robot Action Selection Learning via Layered Dimension Informed Program Synthesis +[link ] +
+Jarrett Holtz, Arjun Guha, and Joydeep Biswas +
+Conference on Robot Learning (CoRL)
Sampling Optimized Code For Type Feedback +[link ] +
+Olivier Flückiger, Andreas Wälchli, Sebastián Krynski, and Jan Vitek +
+Dynamic Languages Symposium (DLS)
Wasm/K: Delimited Continuations For WebAssembly +[link ] +
+Donald Pinckney, Arjun Guha, and Yuriy Brun +
+Dynamic Languages Symposium (DLS)
A Semantics for the Essence of React +[link ] +
+Magnus Madsen, Ondřej Lhoták, and Frank Tip +
+European Conference on Object-Oriented Programming (ECOOP)
Hygienic Macro Technology +[link ] +
+William D. Clinger and Mitchell Wand +
+History of Programming Languages (HOPL)
Model-Based Warp Overlapped Tiling for Image Processing Programs on GPUs +[link ] +
+Abhinav Jangda and Arjun Guha +
+International Conference on Parallel Architectures and Compilation Techniques (PACT)
Typed Dataspace Actors +[link ] +
+Samuel Caldwell, Tony Garnock-Jones, and Matthias Felleisen +
+Journal of Functional Programming (JFP)
A Large-Scale Longitudinal Study of Flaky Tests +[link ] +
+Wing Lam, Stefan Winter, Anjiang Wei, Tao Xie, Darko Marinov, and Jonathan Bell +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
World Age in Julia: Optimizing Method Dispatch in the Presence of Eval +[link ] +
+Julia Belyakova, Benjamin Chung, Jack Gelinas, Jameson Nash, Ross Tate, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Contextual Dispatch for Function Specialization +[link ] +
+Olivier Flückiger, Guido Chari, Ming-Ho Yee, Jan Ječmen, Jakob Hain, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Designing Types for R, Empirically +[link ] +
+Alexi Turcotte, Aviral Goel, Filip Křikava, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Adding Interactive Visual Syntax to Textual Code +[link ] +
+Leif Andersen, Michael Ballantyne, and Matthias Felleisen +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Macros for Domain-Specific Languages +[link ] +
+Michael Ballantyne, Alexis King, and Matthias Felleisen +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
TacTok: Semantics-Aware Proof Synthesis +[link ] +
+Emily First, Yuriy Brun, and Arjun Guha +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Dependent Type Systems as Macros +[link ] +
+Stephen Chang, Michael Ballantyne, Milo Turner, and William J. Bowman +
+Principles of Programming Languages (POPL)
Graduality and Parametricity: Together Again for the First Time +[link ] +
+Max S. New, Dustin Jamner, and Amal Ahmed +
+Principles of Programming Languages (POPL)
Learning Self-Play Agents For Combinatorial Optimization Problems +[link ] +
+Ruiyang Xu and Karl Lieberherr +
+The Knowledge Engineering Review
2019 +
+
Formal Approaches to Secure Compilation: A Survey of Fully Abstract Compilation and Related Work +[link ] +
+Marco Patrignani, Amal Ahmed, and Dave Clarke +
+ACM Computing Surveys (CSUR)
Can Android Run on Time? Extending and Measuring the Android Platform's Timeliness +[link ] +
+Yin Yan, Girish Gokul, Karthik Dantu, Steven Y. Ko, Lukasz Ziarek, and Jan Vitek +
+ACM Transactions on Embedded Computing Systems (TECS)
R Melts Brains: An IR for First-Class Environments and Lazy Effectful Arguments +[link ] +
+Olivier Flückiger, Ming-Ho Yee, Guido Chari, Jakob Hain, Jan Ječmen and Jan Vitek +
+Dynamic Languages Symposium (DLS)
Julia's Efficient Algorithm for Subtyping Unions and Covariant Tuples +[link ] +
+Benjamin Chung, Francesco Zappa Nardelli, and Jan Vitek +
+European Conference on Object-Oriented Programming (ECOOP)
A Study of Call Graph Construction for JVM-Hosted Languages +[link ] +
+Karim Ali, Xiaoni Lai, Zhaoyi Luo, Ondřej Lhoták, Julian Dolby, and Frank Tip +
+IEEE Transactions on Software Engineering (TSE)
The Next 700 Compiler Correctness Theorems (Functional Pearl) +[link ] +
+Daniel Patterson and Amal Ahmed +
+International Conference on Functional Programming (ICFP)
How to Evaluate the Performance of Gradual Type Systems +[link ] +
+Ben Greenman, Asumu Takikawa, Max S. New, Daniel Feltey, Robert Bruce Findler, Jan Vitek, and Matthias Felleisen +
+Journal of Functional Programming (JFP)
Scala Implicits are Everywhere: A Large-Scale Study of the Use of Implicits in the Wild +[link ] +
+Filip Křikava, Heather Miller, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
On the Design, Implementation and Use of Laziness in R +[link ] +
+Aviral Goel and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Complete Monitors for Gradual Types +[link ] +
+Ben Greenman, Matthias Felleisen, and Christos Dimoulas +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Under Control: Compositionally Correct Closure Conversion with Mutable State +[link ] +
+Phillip Mates, Jamie Perconti, and Amal Ahmed +
+Principles and Practice of Declarative Programming (PPDP)
Gradual Type Theory +[link ] +
+Max S. New, Daniel R. Licata, and Amal Ahmed +
+Principles of Programming Languages (POPL)
From Macros to DSLs: The Evolution of Racket +[link ] +
+Ryan Culpepper, Matthias Felleisen, Matthew Flatt, and Shriram Krishnamurthi +
+Summit on Advances in Programming Langugages (SNAPL)
On the Impact of Programming Languages on Code Quality +[link ] +
+Emery D. Berger, Celeste Hollenbeck, Petr Maj, Olga Vitek, and Jan Vitek +
+Transactions on Programming Languages and Systems (TOPLAS)
2018 +
+
The Behavior of Gradual Types: A User Study +[link ] +
+Preston Tunnell Wilson, Ben Greenman, Justin Pombrio, and Shriram Krishnamurthi +
+Dynamic Languages Symposium (DLS)
KafKa: Gradual Typing for Objects + +
+Benjamin Chung, Paley Li, Francesco Zappa Nardelli, and Jan Vitek +
+European Conference on Object-Oriented Programming (ECOOP)
Practical AJAX Race Detection for JavaScript Web Applications +[link ] +
+Christoffer Quist Adamsen, Anders Møller, Saba Alimadadi, and Frank Tip +
+European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)
Soundness of a Concurrent Collector for Actors +[link ] +
+Juliana Franco, Sylvain Clebsch, Sophia Drossopoulou, Jan Vitek, and Tobias Wrigstad +
+European Symposium on Programming (ESOP)
Call-by-name Gradual Type Theory + +
+Max S. New and Daniel R. Licata +
+Formal Structures for Computation and Deduction (FSCD)
Platform-Independent Dynamic Taint Analysis for JavaScript +[link ] +
+Rezwana Karim, Frank Tip, Alena Sochurkova, and Koushik Sen +
+IEEE Transactions on Software Engineering (TSE)
Graduality from Embedding-Projection Pairs + +
+Max S. New and Amal Ahmed +
+International Conference on Functional Programming (ICFP)
Contextual Equivalence for a Probabilistic Language with Continuous Random Variables and Recursion +[link ] +
+Mitchell Wand, Ryan Culpepper, Theophilos Giannakopoulos, and Andrew Cobb +
+International Conference on Functional Programming (ICFP)
A Spectrum of Soundness and Performance + +
+Ben Greenman and Matthias Felleisen +
+International Conference on Functional Programming (ICFP)
Tests from Traces: Automated Unit Test Generation for R + +
+Filip Křikava, Jan Vitek +
+International Symposium on Software Testing and Analysis (ISSTA)
Verifying a Concurrent Garbage Collector with a Rely-Guarantee Methodology +[link ] +
+Yannick Zakowski, David Cachera, Delphine Demange, Gustavo Petri, David Pichardie, Suresh Jagannathan, and Jan Vitek +
+Journal of Automated Reasoning (JAR)
Collapsible Contracts: Fixing a Pathology of Gradual Typing +[link ] +
+Daniel Feltey, Ben Greenman, Christophe Scholliers, Robby Findler, and Vincent St-Amour +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Finding Broken Promises in Asynchronous JavaScript Programs +[link ] +
+Saba Alimadadi, Di Zhong, Magnus Madsen, and Frank Tip +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Julia Subtyping: a Rational Reconstruction +[link ] +
+Francesco Zappa Nardelli, Julia Belyakova, Artem Pelenitsyn, Benjamin Chung, Jeff Bezanson, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Julia: Dynamism and Performance Reconciled by Design +[link ] +
+Jeff Bezanson, Benjamin Chung, Jiahao Chen, Stefan Karpinski, Viral B Shah, Jan Vitek, and Lionel Zoubritzky +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Test Generation for Higher-Order Functions in Dynamic Languages +[link ] +
+Marija Selakovic, Michael Pradel, Rezwana Karim Nawrin, and Frank Tip +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Correctness of Speculative Optimizations with Dynamic Deoptimization +[link ] +
+Olivier Fluckiger, Gabriel Scherer, Ming-Ho Yee, Aviral Goel, Amal Ahmed, and Jan Vitek +
+Principles of Programming Languages (POPL)
Symbolic Types for Lenient Symbolic Execution +[link ] +
+Stephen Chang, Alex Knauth, and Emina Torlak +
+Principles of Programming Languages (POPL)
Type-Preserving CPS Translation of Σ and Π Types is Not Not Possible +[link ] +
+William J. Bowman, Youyou Cong, Nick Rioux, and Amal Ahmed +
+Principles of Programming Languages (POPL)
Simplicitly: Foundations and Applications of Implicit Function Types +[link ] +
+Martin Odersky, Olivier Blanvillain, Fengyun Liu, Aggelos Biboudis, Heather Miller, and Sandro Stucki +
+Principles of Programming Languages (POPL)
Typed Closure Conversion of the Calculus of Constructions +[link ] +
+William J. Bowman and Amal Ahmed +
+Programming Language Design and Implementation (PLDI)
Feature-specific Profiling +[link ] +
+Leif Andersen, Vincent St-Amour, Jan Vitek, and Matthias Felleisen +
+Transactions on Programming Languages and Systems (TOPLAS)
Rank Polymorphism Viewed as a Constraint Problem +[link ] +
+Justin Slepak, Panagiotis Manolios, and Olin Shivers +
+Workshop on Libraries, Languages, and Compilers for Array Programming (ARRAY)
On the Cost of Type-Tag Soundness +[link ] +
+Ben Greenman and Zeina Migeed +
+Workshop on Partial Evaluation and Program Manipulation (PEPM)
Contextual Equivalence for a Probabilistic Language with Continuous Random Variables and Recursion +[link ] +
+Mitchell Wand, Theophilos Giannakopoulos, Andrew Cobb, and Ryan Culpepper +
+Workshop on Probabilistic Programming Semantics (PPS)
2017 +
+
Parallelizing Julia with a Non-invasive DSL +[link ] +
+Todd Anderson, Hai Liu, Lindsey Kuper, Ehsan Totoni, Jan Vitek, and Tatiana Shpeisman +
+European Conference on Object-Oriented Programming (ECOOP)
Contextual Equivalence for Probabilistic Programs with Continuous Random Variables and Scoring +[link ] +
+Ryan Culpepper and Andrew Cobb +
+European Symposium on Programming (ESOP)
Verifying a Concurrent Garbage Collector using a Rely-Guarantee Methodology +[link ] +
+Yannick Zakowski, David Cachera, Delphine Demange, Gustavo Petri, David Pichardie, Suresh Jagannathan, and Jan Vitek +
+Interactive Theorem Proving (ITP)
Inferring Scope through Syntactic Sugar +[link ] +
+Justin Pombrio, Shriram Krishnamurthi, and Mitchell Wand +
+International Conference on Functional Programming (ICFP)
No-Brainer CPS Conversion +[link ] +
+Milo Davis, William Meehan, and Olin Shivers +
+International Conference on Functional Programming (ICFP)
Super 8 Languages for Making Movies (Functional Pearl) +[link ] +
+Leif Andersen, Stephen Chang, and Matthias Felleisen +
+International Conference on Functional Programming (ICFP)
Theorems for Free for Free: Parametricity, With and Without Types +[link ] +
+Amal Ahmed, Dustin Jamner, Jeremy G. Siek, and Philip Wadler +
+International Conference on Functional Programming (ICFP)
Orca: GC and Type System Co-Design for Actor Languages +[link ] +
+Sylvain Clebsch, Juliana Franco, Sophia Drossopoulou, Albert Mingkun Yang, Tobias Wrigstad, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Déj́à Vu: A Map of Code Duplicates on GitHub +[link ] +
+Crista Lopes, Petr Maj, Pedro Martins, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Type Systems as Macros +[link ] +
+Stephen Chang, Alex Knauth, Ben Greenman +
+Principles of Programming Languages (POPL)
Deciding equivalence with sums and the empty type +[link ] +
+Gabriel Scherer +
+Principles of Programming Languages (POPL)
FunTAL: Reasonably Mixing a Functional Language with Assembly +[link ] +
+Daniel Patterson, Jamie Perconti, Christos Dimoulas, and Amal Ahmed +
+Programming Language Design and Implementation (PLDI)
Making Android Run on Time +[link ] +
+Yin Yan, Karthik Dantu, Steven Y. Ko, Jan Vitek, and Lukasz Ziarek +
+Real-time and Embedded Technology and Application Symposium (RTAS)
Linking Types for Multi-Language Software: Have Your Cake and Eat It Too +[link ] +
+Daniel Patterson and Amal Ahmed +
+Summit on Advances in Programming Langugages (SNAPL)
Search for Program Structure +[link ] +
+Gariel Scherer +
+Summit on Advances in Programming Langugages (SNAPL)
Migratory Typing: Ten Years Later +[link ] +
+Sam Tobin-Hochstadt, Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Ben Greenman, Andrew M. Kent, Vincent St-Amour, T. Stephen Strickland, and Asumu Takikawa +
+Summit on Advances in Programming Langugages (SNAPL)
2016 +
+
Coordinated Concurrent Programming in Syndicate +[link ] +
+Tony Garnock-Jones and Matthias Felleisen +
+European Symposium on Programming (ESOP)
Fully Abstract Compilation via Universal Embedding +[link ] +
+Max S. New, William J. Bowman, and Amal Ahmed +
+International Conference on Functional Programming (ICFP)
Oh Lord, Please Don’t Let Contracts Be Misunderstood (Functional Pearl) +[link ] +
+Christos Dimoulas, Max S. New, Robert Bruce Findler, and Matthias Felleisen +
+International Conference on Functional Programming (ICFP)
Is sound gradual typing dead? +[link ] +
+Asumu Takikawa, Daniel Feltey, Ben Greenman, Max S. New, Jan Vitek, and Matthias Felleisen +
+Principles of Programming Languages (POPL)
2015 +
+
Concrete Types for TypeScript +[link ] +
+Gregor Richards, Francesco Zappa Nardelli, and Jan Vitek +
+European Conference on Object-Oriented Programming (ECOOP)
Cooking the Books: Formalizing JMM Implementation Recipes +[link ] +
+Gustavo Petri, Jan Vitek, and Suresh Jagannathan +
+European Conference on Object-Oriented Programming (ECOOP)
Toward practical gradual typing +[link ] +
+Asumu Takikawa, Daniel Feltey, Earl Dean, Matthew Flatt, Robert Bruce Findler, Sam Tobin-Hochstadt, and Matthias Felleisen +
+European Conference on Object-Oriented Programming (ECOOP)
Feature-specific Profiling +[link ] +
+Vincent St-Amour, Leif Andersen, Matthias Felleisen +
+International Conference on Compiler Construction (CC)
Noninterference for Free +[link ] +
+William J. Bowman and Amal Ahmed +
+International Conference on Functional Programming (ICFP)
Repeatability, reproducibility and rigor in CS research +[link ] +
+Jan Vitek +
+PLMW@POPL
Verified Compilers for a Multi-Language World +[link ] +
+Amal Ahmed +
+Summit on Advances in Programming Langugages (SNAPL)
The Racket Manifesto +[link ] +
+Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Shriram Krishnamurthi, Eli Barzilay, Jay McCarthy, Sam Tobin-Hochstadt +
+Summit on Advances in Programming Langugages (SNAPL)
Transferring Skills at Solving Word Problems from Computing to Algebra Through Bootstrap +[link ] +
+Emmanuel Schanzer, Kathi Fisler, Shriram Krishnamurthi, Matthias Felleisen +
+Technical Symposium on Computer Science Education (SIGCSE)
2014 +
+
An Array-Oriented Language with Static Rank Polymorphism +[link ] +
+Justin Slepak, Olin Shivers, and Panagiotis Manolios +
+European Symposium on Programming (ESOP)
Verifying an Open Compiler Using Multi-Language Semantics +[link ] +
+James T. Perconti and Amal Ahmed +
+European Symposium on Programming (ESOP)
The Network as a Language Construct +[link ] +
+Tony Garnock-Jones, Sam Tobin-Hochstadt, and Matthias Felleisen +
+European Symposium on Programming (ESOP)
M3: high-performance memory management from off-the-shelf components +[link ] +
+David Terei, Alex Aiken, and Jan Vitek +
+ISMM
Romeo: a system for more flexible binding-safe programming +[link ] +
+Paul Stansifer and Mitchell Wand +
+International Conference on Functional Programming (ICFP)
Database Queries that Explain their Work +[link ] +
+James Cheney, Amal Ahmed, and Umut Acar +
+Principles and Practice of Declarative Programming (PPDP)
Profiling for Laziness +[link ] +
+Stephen Chang, Matthias Felleisen +
+Principles of Programming Languages (POPL)
Atomicity refinement for verified compilation +[link ] +
+Suresh Jagannathan, Gustavo Petri, Jan Vitek, David Pichardie, and Vincent Laporte +
+Programming Language Design and Implementation (PLDI)
Atomicity Refinement for Verified Compilation +[link ] +
+Suresh Jagannathan, Vincent Laporte, Gustavo Petri, David Pichardie, and Jan Vitek +
+Transactions on Programming Languages and Systems (TOPLAS)
A fast abstract syntax tree interpreter for R +[link ] +
+Tomas Kalibera, Petr Maj, Flor, and Jan Vitek +
+VEE
The case for the three R's of systems research: repeatability, reproducibility and rigor +[link ] +
+Jan Vitek +
+VEE
2013 +
+
Option Contracts +[link ] +
+Christos Dimoulas, Robert Bruce Findler, Matthias Felleisen +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Logical Relations for Fine-Grained Concurrency +[link ] +
+Aaron Turon, Jacob Thamsborg, Amal Ahmed, Lars Birkedal, Derek Dreyer +
+Principles of Programming Languages (POPL)
Contracts for First-Class Classes +[link ] +
+T. Stephen Strickland, Christos Dimoulas, Asumu Takikawa, and Matthias Felleisen +
+Transactions on Programming Languages and Systems (TOPLAS)
2012 +
+
The Call-by-need Lambda Calculus, Revisited +[link ] +
+Stephen Chang and Matthias Felleisen +
+European Symposium on Programming (ESOP)
Complete Monitors for Behavioral Contracts +[link ] +
+Christos Dimoulas, Sam Tobin-Hochstadt, and Matthias Felleisen +
+European Symposium on Programming (ESOP)
Seeing the futures: profiling shared-memory parallel Racket +[link ] +
+James Swaine, Burke Fetscher, Vincent St-Amour, Robby Findler and Matthew Flatt +
+Functional High-Performance Computing (FHPC)
Gradual Typing for First-Class Classes +[link ] +
+Asumu Takikawa, T. Stephen Strickland, Christos Dimoulas, Sam Tobin-Hochstadt, Matthias Felleisen +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Optimization Coaching +[link ] +
+Vincent St-Amour, Sam Tobin-Hochstadt, Matthias Felleisen +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Practical Programming with Substructural Types +[link ] +
+Jesse A. Tov +
+PhD Dissertation, Northeastern University
Typing the Numeric Tower +[link ] +
+Vincent St-Amour, Sam Tobin-Hochstadt, Matthew Flatt, and Matthias Felleisen +
+Practical Aspects of Declarative Languages (PADL)
Run Your Research +[link ] +
+Casey Klein, John Clements, Christos Dimoulas, Carl Eastlund, Matthias Felleisen, Matthew Flatt, Jay McCarthy, Jon Rafkind, Sam Tobin-Hochstadt, Robert Bruce Findler +
+Principles of Programming Languages (POPL)
2011 +
+
Bounded-latency regional garbage collection +[link ] +
+Felix S. Klock II and William D. Clinger +
+Dynamic Languages Symposium (DLS)
From Stack Traces to Lazy Rewriting Sequences +[link ] +
+Stephen Chang, Eli Barzilay, John Clements, Matthias Felleisen +
+Implementation and Application of Functional Languages (IFL)
Modular rollback through control logging: a pair of twin functional pearls +[link ] +
+Olin Shivers and Aaron Joseph Turon +
+International Conference on Functional Programming (ICFP)
Pushdown flow analysis of first-class control +[link ] +
+Dimitrios Vardoulakis and Olin Shivers +
+International Conference on Functional Programming (ICFP)
Parsing reflective grammars +[link ] +
+Paul Stansifer and Mitchell Wand +
+LDTA
A Resource Analysis of the π-calculus +[link ] +
+Aaron Joseph Turon and Mitchell Wand +
+Mathematical Foundations of Programming Semantics (MFPS)
A Theory of Substructural Types and Control +[link ] +
+Jesse A. Tov and Riccardo Pucella +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Ordering multiple continuations on the stack +[link ] +
+Dimitrios Vardoulakis and Olin Shivers +
+PEPM
Practical Affine Types +[link ] +
+Jesse A. Tov and Riccardo Pucella +
+Principles of Programming Languages (POPL)
Correct Blame for Contracts: No More Scapegoating +[link ] +
+Christos Dimoulas, Robert Bruce Findler, Cormac Flanagan, Matthias Felleisen +
+Principles of Programming Languages (POPL)
A separation logic for refining concurrent objects +[link ] +
+Aaron Joseph Turon and Mitchell Wand +
+Principles of Programming Languages (POPL)
Languages as Libraries +[link ] +
+Sam Tobin-Hochstadt, Vincent St-Amour, Ryan Culpepper, Matthew Flatt, Matthias Felleisen +
+Programming Language Design and Implementation (PLDI)
A Family of Abstract Interpretations for Static Analysis of Concurrent Higher-Order Programs +[link ] +
+Matthew Might and David Van Horn +
+The 18th International Static Analysis Symposium
On Contract Satisfaction in a Higher-Order World +[link ] +
+Christos Dimoulas, Matthias Felleisen +
+Transactions on Programming Languages and Systems (TOPLAS)
2010 +
+
Weaving Generic Programming and Traversal Performance +[link ] +
+Bryan Chadwick and Karl Lieberherr +
+AOSD
Contracts for First-Class Classes +[link ] +
+T. Stephen Strickland, Matthias Felleisen +
+Dynamic Languages Symposium (DLS)
Stateful Contracts for Affine Types +[link ] +
+Jesse A. Tov and Riccardo Pucella +
+European Symposium on Programming (ESOP)
CFA2: A Context-Free Approach to Control-Flow Analysis +[link ] +
+Dimitrios Vardoulakis and Olin Shivers +
+European Symposium on Programming (ESOP)
Bottom-up beta-reduction: Uplinks and lambda-DAGs +[link ] +
+Olin Shivers and Mitchell Wand +
+Fundamenta Informaticae
Abstracting Abstract Machines +[link ] +
+David Van Horn and Matthew Might +
+International Conference on Functional Programming (ICFP)
Fortifying Macros +[link ] +
+Ryan Culpepper, Matthias Felleisen +
+International Conference on Functional Programming (ICFP)
Logical Types for Untyped Languages +[link ] +
+Sam Tobin-Hochstadt, Matthias Felleisen +
+International Conference on Functional Programming (ICFP)
TeachScheme!---A Checkpoint (Abstract) +[link ] +
+Matthias Felleisen +
+International Conference on Functional Programming (ICFP)
CFA2: a Context-Free Approach to Control-Flow Analysis +[link ] +
+Dimitrios Vardoulakis and Olin Shivers +
+Northeastern University College of Computer and Information Science Technical Reports
Functional Adaptive Programming +[link ] +
+Bryan Chadwick +
+PhD Dissertation, Northeastern University
Resolving and Exploiting the k-CFA Paradox +[link ] +
+Matthew Might, Yannis Smaragdakis and David Van Horn +
+Programming Language Design and Implementation (PLDI)
Pushdown Control-Flow Analysis of Higher-Order Programs +[link ] +
+Christopher Earl, Matthew Might and David Van Horn +
+Scheme and Functional Programming Workshop
Evaluating Call By Need on the Control Stack +[link ] +
+Stephen Chang, David Van Horn and Matthias Felleisen +
+Trends in Functional Programming (TFP)
Hygienic Macros for ACL2 +[link ] +
+Carl Eastlund, Matthias Felleisen +
+Trends in Functional Programming (TFP)
Adding Types to Untyped Languages (Abstract) +[link ] +
+Matthias Felleisen +
+Types in Language Design and Implementation (TLDI)
Algorithms for Traversal-Based Generic Programming +[link ] +
+Bryan Chadwick and Karl Lieberherr +
+Workshop on Generic Programming
2009 +
+
Automatic Verification for Interactive Graphical Programs +[link ] +
+Carl Eastlund and Matthias Felleisen +
+ACL2 Workshop
Contracts for First-Class Modules +[link ] +
+T. Stephen Strickland, Matthias Felleisen +
+Dynamic Languages Symposium (DLS)
Practical Variable-Arity Polymorphism +[link ] +
+T. Stephen Strickland, Sam Tobin-Hochstadt, and Matthias Felleisen +
+European Symposium on Programming (ESOP)
A Type System for Functional Traversal-Based Aspects +[link ] +
+Bryan Chadwick and Karl Lieberherr +
+FOAL Workshop
Nested and Dynamic Contract Boundaries +[link ] +
+T. Stephen Strickland, Matthias Felleisen +
+Implementation and Application of Functional Languages (IFL)
A Functional I/O System (or Fun for Freshman Kids) +[link ] +
+Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Shriram Krishnamurthi +
+International Conference on Functional Programming (ICFP)
Regular expression derivatives reexamined +[link ] +
+Scott Owens, John Reppy and Aaron Turon +
+Journal of Functional Programming (JFP)
Toward a Practical Module System for ACL2 +[link ] +
+Carl Eastlund and Matthias Felleisen +
+Practical Aspects of Declarative Languages (PADL)
Making Induction Manifest in Modular ACL2 +[link ] +
+Carl Eastlund, Matthias Felleisen +
+Principles and Practice of Declarative Programming (PPDP)
Future Contracts +[link ] +
+Christos Dimoulas, Riccardo Pucella, Matthias Felleisen +
+Principles and Practice of Declarative Programming (PPDP)
Sequence Traces for Object-Oriented Executions +[link ] +
+Carl Eastlund, Matthias Felleisen +
+Scheme and Functional Programming Workshop
Ryan Culpepper, Matthias Felleisen +[link ] +
+Debugging Hygienic Macros +
+Science of Computer Programming
All-Terimation(T) +[link ] +
+Panagiotis Manolios and Aaron Turon +
+TACAS
The Higher-Order Aggregate Update Problem +[link ] +
+Christos Dimoulas and Mitchell Wand +
+Verification, Model Checking, and Abstract Interpretation (VMCAI)
2008 +
+
Variable-Arity Generic Interfaces +[link ] +
+T. Stephen Strickland and Richard Cobbe and Matthias Felleisen +
+(tech report) Northeastern University College of Computer and Information Science no. NU-CCIS-08-01
A Compositional Trace Semantics for Orc +[link ] +
+Dimitrios Vardoulakis and Mitchell Wand +
+COORDINATION
A Theory of Hygienic Macros +[link ] +
+David Herman and Mitchell Wand +
+European Symposium on Programming (ESOP)
Functional Programming and Theorem Proving for Undergraduates: A Progress Report +[link ] +
+Rex Page, Carl Eastlund, and Matthias Felleisen +
+Functional and Declarative Programming in Education (FDPE)
Trusted Theorem Proving: A Case Study in SLD-Resolution +[link ] +
+Konstantine Arkoudas and Olin Shivers +
+ISoLA
Building language towers with Ziggurat +[link ] +
+David Fisher and Olin Shivers +
+Journal of Functional Programming (JFP)
Exploiting reachability and cardinality in higher-order flow analysis +[link ] +
+Matthew Might and Olin Shivers +
+Journal of Functional Programming (JFP)
Essentials of programming languages (3. ed.) +[link ] +
+Daniel P. Friedman and Mitchell Wand +
+MIT Press
Much Ado about Nothing: Putting Java's Null in its Place +[link ] +
+Richard Cobbe +
+PhD Dissertation, Northeastern University
The Design and Implementation of Typed Scheme +[link ] +
+Sam Tobin-Hochstadt, Matthias Felleisen +
+Principles of Programming Languages (POPL)
Programming languages: fundamental concepts for expanding and disciplining the mind +[link ] +
+Mitchell Wand and Daniel P. Friedman +
+SIGPLAN Notices
Why teach programming languages +[link ] +
+Olin Shivers +
+SIGPLAN Notices
Haskell Session Types with (Almost) No Class +[link ] +
+Riccardo Pucella and Jesse A. Tov +
+SIGPLAN Symposium on Haskell
Caml-Shcaml: An OCaml Library for Unix Shell Programming +[link ] +
+Alec Heller and Jesse A. Tov +
+SIGPLAN workshop on ML
2007 +
+
ACL2 for Freshmen: First Experiences +[link ] +
+Carl Eastlund, Dale Vaillancourt, Matthias Felleisen +
+ACL2 Workshop
Debugging Macros +[link ] +
+Ryan Culpepper, Matthias Felleisen +
+Generative Programming: Concepts & Experience (GPCE)
On the correctness of the Krivine machine +[link ] +
+Mitchell Wand +
+Higher-Order and Symbolic Computation
Implementation and Use of the PLT Scheme Web Server +[link ] +
+Shriram Krishnamurthi, Peter Walton Hopkins, Jay McCarthy, Paul T. Graunke, Greg Pettyjohn, Matthias Felleisen +
+Higher-Order and Symbolic Computation
Adding Delimited and Composable Control to a Production Programming Environment +[link ] +
+Matthew Flatt, Gang Yu, Robert Bruce Findler, Matthias Felleisen +
+International Conference on Functional Programming (ICFP)
Functional Pearl: The Great Escape. Or, How to Jump the Border Without Getting Caught +[link ] +
+David Herman +
+International Conference on Functional Programming (ICFP)
Advanced Macrology and the Implementation of Typed Scheme +[link ] +
+Ryan Culpepper, Sam Tobin-Hochstadt and Matthew Flatt +
+Scheme Workshop
Analyzing the environment structure of higher-order languages using frame strings +[link ] +
+Matthew Might and Olin Shivers +
+Theoretical Computer Science
Space-Efficient Gradual Typing +[link ] +
+David Herman, Aaron Tomb and Cormac Flanagan +
+Trends in Functional Programming (TFP)
Model Checking Via GammaCFA +[link ] +
+Matthew Might, Benjamin Chambers, and Olin Shivers +
+Verification, Model Checking, and Abstract Interpretation (VMCAI)
Status Report: Specifying JavaScript with ML +[link ] +
+David Herman and Cormac Flanagan +
+Workshop on ML
2006 +
+
Bisimulations for Untyped Imperative Objects +[link ] +
+Vasileios Koutavas and Mitchell Wand +
+European Symposium on Programming (ESOP)
Improving flow analyses via GammaCFA: abstract garbage collection and counting +[link ] +
+Matthew Might and Olin Shivers +
+International Conference on Functional Programming (ICFP)
Static analysis for syntax objects +[link ] +
+David Fisher and Olin Shivers +
+International Conference on Functional Programming (ICFP)
Multi-return function call +[link ] +
+Olin Shivers and David Fisher +
+Journal of Functional Programming (JFP)
Small bisimulations for reasoning about higher-order imperative programs +[link ] +
+Vasileios Koutavas and Mitchell Wand +
+Principles of Programming Languages (POPL)
Environment analysis via Delta CFA +[link ] +
+Matthew Might and Olin Shivers +
+Principles of Programming Languages (POPL)
Continuations and transducer composition +[link ] +
+Olin Shivers and Matthew Might +
+Programming Language Design and Implementation (PLDI)
Linear combinations of radioactive decay models for generational garbage collection +[link ] +
+William D. Clinger and Fabio V. Rojas +
+Science of Computer Programming
2005 +
+
Selectors Make Set-Based Analysis Too Hard +[link ] +
+Philippe Meunier, Robert Bruce Findler, Paul Steckler, and Mitchell Wand +
+Higher-Order and Symbolic Computation
2004 +
+
Relating models of backtracking +[link ] +
+Mitchell Wand and Dale Vaillancourt +
+International Conference on Functional Programming (ICFP)
A semantics for advice and dynamic join points in aspect-oriented programming +[link ] +
+Mitchell Wand, Gregor Kiczales, and Christopher Dutchyn +
+Transactions on Programming Languages and Systems (TOPLAS)
2003 +
+
Understanding aspects: extended abstract +[link ] +
+Mitchell Wand +
+International Conference on Functional Programming (ICFP)
CPS transformation of flow information +[link ] +
+Jens Palsberg and Mitchell Wand +
+Journal of Functional Programming (JFP)
2002 +
+
A Modular, Extensible Proof Method for Small-Step Flow Analyses +[link ] +
+Mitchell Wand and Galen B. Williamson +
+European Symposium on Programming (ESOP)
An experimental study of renewal-older-first garbage collection +[link ] +
+Lars Thomas Hansen and William D. Clinger +
+International Conference on Functional Programming (ICFP)
Concurrent Remembered Set Refinement in Generational Garbage Collection +[link ] +
+David Detlefs, Ross Knippel, William D. Clinger, and Matthias Jacob +
+Java Virtual Machine Research and Technology Symposium
2001 +
+
Set constraints for destructive array update optimization +[link ] +
+Mitchell Wand and William D. Clinger +
+Journal of Functional Programming (JFP)
Essentials of programming languages (2. ed.) +[link ] +
+Daniel P. Friedman, Mitchell Wand, and Christopher T. Haynes +
+MIT Press
A Semantics for Advice and Dynamic Join Points in Aspect-Oriented Programming +[link ] +
+Mitchell Wand +
+SAIG
2000 +
+
Analysis-based program transformations +[link ] +
+Mitchell Wand +
+ACM SIGSOFT Software Engineering Notes
Optimizing memory usage in higher-order programming languages: theoretical and experimental studies +[link ] +
+Mitchell Wand and William D. Clinger +
+ACM SIGSOFT Software Engineering Notes
1999 +
+
Implementation Strategies for First-Class Continuations +[link ] +
+William D. Clinger, Anne Hartheimer, and Eric Ost +
+Higher-Order and Symbolic Computation
Continuation-Based Multiprocessing Revisited +[link ] +
+Mitchell Wand +
+Higher-Order and Symbolic Computation
Continuation-Based Multiprocessing +[link ] +
+Mitchell Wand +
+Higher-Order and Symbolic Computation
Trampolined Style +[link ] +
+Steven E. Ganz, Daniel P. Friedman, and Mitchell Wand +
+International Conference on Functional Programming (ICFP)
A Language for Specifying Recursive Traversals of Object Structures +[link ] +
+Johan Ovlinger and Mitchell Wand +
+Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)
Constraint Systems for Useless Variable Elimination +[link ] +
+Mitchell Wand and Igor Siveroni +
+Principles of Programming Languages (POPL)
1998 +
+
Revised Report on the Algorithmic Language Scheme +[link ] +
+Harold Abelson, R. Kent Dybvig, Christopher T. Haynes, Guillermo Juan Rozas, N. I. Adams IV, Daniel P. Friedman, Eugene E. Kohlbecker, Guy L. Steele Jr., David H. Bartley, Robert H. Halstead Jr., Don Oxley, Gerald J. Sussman, G. Brooks, Chris Hanson, Kent M. Pitman, and Mitchell Wand +
+Higher-Order and Symbolic Computation
Set Constraints for Destructive Array Update Optimization +[link ] +
+Mitchell Wand and William D. Clinger +
+ICCL
The Theory of Fexprs is Trivial +[link ] +
+Mitchell Wand +
+Lisp and Symbolic Computation
Proper Tail Recursion and Space Efficiency +[link ] +
+William D. Clinger +
+Programming Language Design and Implementation (PLDI)
1997 +
+
Type Inference with Non-Structural Subtyping +[link ] +
+Jens Palsberg, Mitchell Wand, and Patrick O'Keefe +
+Formal Aspects of Computing
Denotational Semantics Using an Operationally-Based Term Model +[link ] +
+Mitchell Wand and Gregory T. Sullivan +
+Principles of Programming Languages (POPL)
Generational Garbage Collection and the Radioactive Decay Model +[link ] +
+William D. Clinger and Lars Thomas Hansen +
+Programming Language Design and Implementation (PLDI)
Lightweight Closure Conversion +[link ] +
+Paul Steckler and Mitchell Wand +
+Transactions on Programming Languages and Systems (TOPLAS)
1996 +
+
Untyped Lambda-Calculus with Input-Output +[link ] +
+Jerzy Tiuryn and Mitchell Wand +
+CAAP
Compiler Correctness for Concurrent Languages +[link ] +
+David S. Gladstein and Mitchell Wand +
+COORDINATION
Modeling Subobject-based Inheritance +[link ] +
+Jonathan G. Rossie Jr., Daniel P. Friedman, and Mitchell Wand +
+European Conference on Object-Oriented Programming (ECOOP)
1995 +
+
Compiler Correctness for Parallel Languages +[link ] +
+Mitchell Wand +
+FPCA
VLISP: A Verified Implementation of Scheme +[link ] +
+Joshua D. Guttman, John D. Ramsdell, and Mitchell Wand +
+Lisp and Symbolic Computation
The VLISP Verified PreScheme Compiler +[link ] +
+Dino Oliva, John D. Ramsdell, and Mitchell Wand +
+Lisp and Symbolic Computation
Strong Normalization with Non-Structural Subtyping +[link ] +
+Mitchell Wand, Patrick O'Keefe, and Jens Palsberg +
+Mathematical Structures in Computer Science
1994 +
+
Conditional Lambda-Theories and the Verification of Static Properties of Programs +[link ] +
+Mitchell Wand and Zheng-Yu Wang +
+Information and Computation
Selective and Lightweight Closure Conversion +[link ] +
+Mitchell Wand and Paul Steckler +
+Principles of Programming Languages (POPL)
Selective Thunkification +[link ] +
+Paul Steckler and Mitchell Wand +
+SAS
1993 +
+
Specifying the Correctness of Binding-Time Analysis +[link ] +
+Mitchell Wand +
+Journal of Functional Programming (JFP)
Specifying the Correctness of Binding-Time Analysis +[link ] +
+Mitchell Wand +
+Principles of Programming Languages (POPL)
Type Reconstruction with Recursive Types and Atomic Subtyping +[link ] +
+Jerzy Tiuryn and Mitchell Wand +
+TAPSOFT
1992 +
+
Type Inference for Partial Types is Decidable +[link ] +
+Patrick O'Keefe and Mitchell Wand +
+European Symposium on Programming (ESOP)
Proving the Correctness of Storage Representations +[link ] +
+Mitchell Wand and Dino Oliva +
+LISP and Functional Programming (LFP)
Essentials of programming languages +[link ] +
+Daniel P. Friedman, Mitchell Wand, and Christopher T. Haynes +
+MIT Press
1991 +
+
Automatic Dimensional Inference +[link ] +
+Mitchell Wand and Patrick O'Keefe +
+Computational Logic - Essays in Honor of Alan Robinson
Type Inference for Record Concatenation and Multiple Inheritance +[link ] +
+Mitchell Wand +
+Information and Computation
Correctness of Procedure Representations in Higher-Order Assembly Language +[link ] +
+Mitchell Wand +
+Mathematical Foundations of Programming Semantics (MFPS)
Correctness of Static Flow Analysis in Continuation Semantics +[link ] +
+Margaret Montenyohl and Mitchell Wand +
+Science of Computer Programming
1990 +
+
A Short Proof of the Lexical Addressing Algorithm +[link ] +
+Mitchell Wand +
+Information Processing Letters
Conditional Lambda-Theories and the Verification of Static Properties of Programs +[link ] +
+Mitchell Wand and Zheng-Yu Wang +
+Logic in Computer Science (LICS)
1989 +
+
On the Complexity of Type Inference with Coercion +[link ] +
+Mitchell Wand and Patrick O'Keefe +
+FPCA
Incorporating Static Analysis in a Combinator-Based Compiler +[link ] +
+Margaret Montenyohl and Mitchell Wand +
+Information and Computation
Type Inference for Record Concatenation and Multiple Inheritance +[link ] +
+Mitchell Wand +
+Logic in Computer Science (LICS)
1988 +
+
Abstract Continuations: A Mathematical Semantics for Handling Full Jumps +[link ] +
+Matthias Felleisen, Mitchell Wand, Daniel P. Friedman, and Bruce F. Duba +
+LISP and Functional Programming (LFP)
The Mystery of the Tower Revealed: A Nonreflective Description of the Reflective Tower +[link ] +
+Mitchell Wand and Daniel P. Friedman +
+Lisp and Symbolic Computation
Corrigendum: Complete Type Inference for Simple Objects +[link ] +
+Mitchell Wand +
+Logic in Computer Science (LICS)
Correct Flow Analysis in Continuation Semantics +[link ] +
+Margaret Montenyohl and Mitchell Wand +
+Principles of Programming Languages (POPL)
1987 +
+
Complete Type Inference for Simple Objects +[link ] +
+Mitchell Wand +
+Logic in Computer Science (LICS)
Macro-by-Example: Deriving Syntactic Transformations from their Specifications +[link ] +
+Eugene E. Kohlbecker and Mitchell Wand +
+Principles of Programming Languages (POPL)
Linear Future Semantics and Its Implementation +[link ] +
+Stefan K and Mitchell Wand +
+Science of Computer Programming
1986 +
+
Obtaining Coroutines with Continuations +[link ] +
+Christopher T. Haynes, Daniel P. Friedman, and Mitchell Wand +
+Computer Languages
The Mystery of the Tower Revealed: A Non-Reflective Description of the Reflective Tower +[link ] +
+Mitchell Wand and Daniel P. Friedman +
+LISP and Functional Programming (LFP)
Finding the Source of Type Errors +[link ] +
+Mitchell Wand +
+Principles of Programming Languages (POPL)
1985 +
+
Continuation Semantics in Typed Lambda-Calculi (Summary) +[link ] +
+Albert R. Meyer and Mitchell Wand +
+Logic of Programs
Embedding Type Structure in Semantics +[link ] +
+Mitchell Wand +
+Principles of Programming Languages (POPL)
From interpreter to compiler: a representational derivation +[link ] +
+Mitchell Wand +
+Programs as Data Objects
+

+

© Copyright Programming Research Laboratory 2015-2021 | made by Catchexception s.r.o. | source on GitHub

+top
+ + + + + + + diff --git a/seminars.html b/seminars.html new file mode 100644 index 00000000..610d1d59 --- /dev/null +++ b/seminars.html @@ -0,0 +1,1949 @@ + + + + + + + + + + +Seminars - Programming Research Laboratory - Northeastern University + + + + + + + + + + + + +

Seminars

+ +

Programming Language Seminar

+

The PL Seminar generally meets +in WVH 366 +and on Zoom +on Fridays from 11:00am-12:00pm (ET). +The mailing list is public. +A calendar +(HTML + | iCal) +is available for your convenience. (Note: We temporarily switched to a new calendar at the start of Fall 2021, but we reverted that change. This link is current!)

+ +
+ +

Upcoming Seminars

+ +
+

Past Seminars

+
Making CPython Faster: It's interpreters all the way down! +
+Ken Jin +National University of Singapore + +
+21 May, 2024 12:00PM +WVH 366 +
+

Abstract

+

Python is a popular programming language used worldwide. CPython is its reference implementation, written in C. There has been a multi-year effort to speed up CPython, starting from version 3.11. In this talk, I'll cover what we've done. Specifically, this talk will cover specializing/quickening interpreters, inline caching, JIT compilers, abstract interpretation, and DSLs (domain-specific languages). The key takeaway from this talk is how CPython uses DSLs and abstract interpretation all the way down to reduce maintenance burden and developer error.

+

Bio

+

Ken Jin is a Python/CPython core developer and a student at the National University of Singapore. He is also a part-time software engineer at Quansight Labs. He recently wrote CPython's Tier 2 optimizer (JIT optimizer) in CPython 3.13. At Quansight Labs, he previously worked for a brief period of time on TorchDynamo, PyTorch 2's JIT compiler, and is now working on CPython performance work in multi-threaded environments.


+ +
Building the Tools to Program a Quantum Computer +
+Charles Yuan +MIT +https://people.csail.mit.edu/chenhuiy/ +
+14 May, 2024 12:00PM +WVH 366 +
+

Abstract

+

Bringing the promise of quantum computation into reality requires not only building a quantum computer but also correctly programming it to run a quantum algorithm. To obtain asymptotic advantage over classical algorithms, quantum algorithms rely on the ability of data in quantum superposition to exhibit phenomena such as interference and entanglement. In turn, an implementation of the algorithm as a program must correctly orchestrate these phenomena in the states of qubits. Otherwise, the algorithm would yield incorrect outputs or lose its computational advantage.

+

Given a quantum algorithm, what are the challenges and costs to realizing it as an executable program? In this talk, I answer this question by showing how basic programming abstractions – such as data structures and control flow – upon which many quantum algorithms rely can fail to work correctly or efficiently on a quantum computer. I then show how we can leverage insights from programming languages to re-invent these abstractions to meet the demands of quantum algorithms. This approach holds out a promise of expressive and efficient tools for programming a quantum computer.

+

Bio

+

Charles Yuan is a Ph.D. student at MIT CSAIL working with Prof. Michael Carbin. His current research examines the challenges of programming quantum computers and other emerging models of computation. His work has appeared in the ACM SIGPLAN POPL, OOPSLA, and PLDI conferences and has been recognized with the SIGPLAN Distinguished Artifact Award and the CQE-LPS Doc Bedard Fellowship.


+ +
+

Abstract

+

WebAssembly (Wasm) is a simple, formally specified, low-level bytecode format that is designed to be a good compilation target for statically-typed languages such as C, C++, and Rust (among others). It has excellent sandboxing properties, and has thus seen adoption in applications where untrusted user code must run. Unfortunately, the story for dynamically-typed (and otherwise highly dynamic) languages such as JavaScript running on Wasm-based platforms is less well-developed: the state of the art is to embed an interpreter within a Wasm module and package it with bytecode.

+

In this talk, I walk through (i) background on Wasm-based platforms and why they are attractive; (ii) difficulties that Wasm, as a highly static environment, presents to languages traditionally accelerated by a JIT compiler; and (iii) outline one path to achieve ahead-of-time compilation of such languages to Wasm modules without JIT.

+

The key insight of this work is that WebAssembly provides a novel opportunity to apply partial evaluation to get a compiler "for free" out of an interpreter. I will outline how we first developed and upstreamed a new high-performance interpreter tier in the SpiderMonkey JS engine, then developed a partial evaluator called "weval" for Wasm that, together with the upstreamed interpreter with minimal changes, produces fully ahead-of-time-compiled code from JS source, encapsulating all remaining dynamism in the inline-cache mechanism. This results in a ~2x speedup overall, with ~4x on some microbenchmarks and potential for higher.

+

Bio

+

Chris is a software engineer at Fastly, where he works on WebAssembly-related technology involving Cranelift. Previously, he was at Mozilla, where he worked on Firefox's SpiderMonkey JS just-in-time compiler and Cranelift. Before joining Mozilla, he earned a PhD at Carnegie Mellon University, where he studied compilers, static and dynamic analysis, and before that, CPU microarchitecture.


+ +
Separation Logics for Probabilistic Programs +
+Justin Hsu +Cornell University +https://www.justinhsu.net/ +
+7 April, 2023 12:00PM +Kariotis 302 and Zoom +
+

Abstract

+

Separation is a powerful concept when verifying programs that +manipulate memory, and concurrent programs. In the probabilistic setting, there +are also natural notions of separation. For instance, probabilistic independence +is useful for describing the result of random sampling---a basic operation in +all probabilistic languages---and for reasoning about groups of random +variables. Nevertheless, existing verification methods handle independence +poorly, if at all.

+

I will survey our recent work on probabilistic separation logics, based on +probabilistic models of the logic of bunched implication (BI) and other bunched +logics. In the program logic PSL, separation models probabilistic independence; +we have used PSL to verify information-theoretic security of some well-known +cryptographic constructions. I'll also describe extensions to reason about other +separation-like relations between random variables, like conditional +independence and negative association.

+

Bio

+

Justin Hsu is an assistant professor of Computer Science at Cornell University. +He was previously an assistant professor at UW--Madison and a postdoc at Cornell +and UCL, after receiving his doctorate in Computer Science from the University +of Pennsylvania. His research focuses on formal verification of probabilistic +programs, including algorithms from differential privacy, protocols from +cryptography, and mechanisms from game theory. His work has been recognized by +an NSF CAREER award, a SIGPLAN John C. Reynolds Doctoral Dissertation award, +distinguished paper awards from POPL and CAV, and industrial awards from +Facebook.


+ +
Leapfrog: Certified Equivalence for Protocol Parsers +
+Ryan Doenges +Cornell University +https://ryandoeng.es/ +
+13 March, 2023 12:00PM +Kariotis 302 and Zoom +
+

Abstract

+

In this talk I will describe some recent work on the P4 programming language, +a domain-specific language for programming networking accelerators. First, +I will describe Petr4 (pron. "petra"), our semantics for P4, and give a brief +overview of the language's syntax, type system, and operational semantics. Then +I will cover Leapfrog, a Coq-based framework for verifying the equivalence of +packet parsers defined in P4. Leapfrog is based on P4 Automata (P4A), an +automata-theoretic model based on Petr4's semantics of packet parsers. We +provide a decision procedure for P4A equivalence based on symbolic bisimilarity; +it is implemented in Coq as a tactic and proved sound with respect to the +operational semantics of P4A. The tactic relies on a trusted SAT solver to +decide symbolic entailments. We find the bisimilarity algorithm generalizes to +certain relational properties beyond mere equivalence.

+

Bio

+

Ryan Doenges is a Ph.D. candidate at Cornell University, where he +studies programming language principles for emerging networking devices. +He is advised by Nate Foster.


+ +
Session Types for Information Flow Control and a Logical Relation for Noninterference +
+Stephaie Balzer +Carnegie Mellon University +https://www.cs.cmu.edu/~balzers/ +
+18 November, 2022 12:00PM +WVF 118 and Zoom +
+

Abstract

+

Noninterference guarantees that an attacker cannot infer secrets by interacting with a program. +An information flow control (IFC) type system asserts noninterference by tracking the level of information learned and +disallowing leakage to parties of lesser or unrelated level. In this talk, I explore session types as an enabler for +tracking the flow of information. Session types stipulate according to which protocol messages must be exchanged along +channels connecting concurrently executing processes (or threads). I develop an IFC type system for linear session-typed process +calculus that employs possible worlds from hybrid logic to reason about security levels. To prove that well-typed programs in the +resulting language ensure noninterference, I develop a logical relation. I survey what the main challenges in the development +are --- non-termination (because of general recursive types) and non-determinism (because of concurrency) --- and explain how to +address them. Distinguishing features of the resulting logical relation are its interactive nature (the program interacts with its context) +and the use of an observation index (as opposed to a step index or later modality). The latter ensures compositionality despite concurrency.

+

Bio

+

Stephanie Balzer is an Assistant Research Professor in the Principles of Programming group at CMU. +Her research is concerned with compositional verification of concurrent programs, using logic and type +systems, logical relations, and verification logics.


+ +
Checking Data-Race Freedom of GPU Kernels, Compositionally +
+Tiago Cogumbreiro +University of Massachusetts, Boston +https://cogumbreiro.github.io/ +
+4 November, 2022 12:00PM +WVF 118 and Zoom +
+

Abstract

+

GPUs offer parallelism as a commodity, but they are difficult to programcorrectly. Static analyzers that guarantee data-race freedom + (DRF) areessential to help programmers establish the correctness of theirprograms (kernels). However, existing approaches produce too +many false alarms and struggle to handle larger programs. To address +these limitations we formalize a novel compositional analysis for DRF, +based on access memory protocols. These protocols are behavioral types +that codify the way threads interact over shared memory. +Our work includes fully mechanized proofs of our theoretical results, the +first mechanized proofs in the field of DRF analysis for GPU kernels. +Our theory is implemented in Faial, a tool that outperforms the state-of- +the-art. Notably, it can correctly verify at least 1.42× more real-world +kernels, and it exhibits a linear growth in 4 out of 5 experiments, while +others grow exponentially in all 5 experiments

+

This talk presents the CAV21 publication and reports updates on this research (journal and workshop publications)

+

Bio

+

Tiago Cogumbreiro joined the Department of Computer Science at the University of +Massachusetts Boston in fall 2018 as an assistant professor. Prior to that, +Tiago was a postdoctoral researcher at the Georgia Institute of Technology and +Rice University, supervised by Professor Vivek Sarkar, and a research assistant +at Imperial College London, supervised by Professor Nobuko Yoshida. Tiago +obtained a PhD from the University of Lisbon, Portugal.


+ +
Val Programming Language +
+Dimitri Racordon +Northeastern University +https://kyouko-taiga.github.io/ +
+21 October, 2022 12:00PM +WVF 118 and Zoom +
+

Abstract

+

Val is a zero-cost abstraction language based on two of the best ideas behind C++: generic programming and first-class value semantics. Unlike C++, however, Val achieves memory safety without resorting to defensive copying.

+

In this talk, I will explain how Val codifies best practices from modern C++ into a type system that guarantees type and memory safety while being able to express 99% of the low-level bit pushing one can write in C++ with the same predictable efficiency. I will also attempt to convince you that you don't need neither pointers, references, nor sophisticated lifetime annotations to get there.

+

Bio

+

+ +
Compiler Infrastructure for Accelerator Generators +
+Rachit Nigam +Cornell University +rachitnigam.com +
+7 April, 2022 1:30PM +WVH 366 and Zoom +
+

Abstract

+

Specialized, application-specific hardware accelerators are chipping away at the dominance of traditional, general-purpose CPUs. We need to make it possible for domain experts—not just hardware experts—to harness the efficiency of hardware specialization for the computations they care about. Domain-specific languages (DSLs) for building hardware accelerators offer a way to raise the level of abstraction in hardware design from gates, wires, and clock cycles. Unfortunately, building a new hardware DSL is a gargantuan task requiring not only the design of new abstractions, but also supporting tools such as an optimizing compiler, testing, and debugging infrastructure.

+

Our solution is Calyx (calyxir.org), an intermediate language and a compiler infrastructure that can represent, optimize, and lower accelerators to synthesizable hardware designs. By targeting Calyx instead of a traditional hardware design language, designers can build new DSLs and generate a custom hardware accelerator without needing hardware expertise.

+

Bio

+Rachit Nigam (rachitnigam.com) is PhD candidate at Cornell University interested in programming languages, computer architectures, and compilers that turn programs to architectures.

+ +
PaSh: Light-touch Data-Parallel Shell Processing +
+Konstantinos Kallas +University of Pennsylvania +https://angelhof.github.io/ +
+25 February, 2022 11:00AM +Zoom Only +
+

Abstract

+

"In this talk I will present PaSh, a system that exposes data parallelism in POSIX shell scripts. To achieve that, PaSh offers: (i) an order-aware dataflow model that precisely captures a fragment of the shell, (ii) a set of dataflow transformations that extract parallelism and have been proven to be correct, and (iii) a lightweight framework that captures the correspondence of shell commands and order-aware dataflow nodes. PaSh achieves orders-of-magnitude performance improvements on a variety of scripts found in the wild. PaSh is open-source https://github.com/binpash/pash and hosted under the Linux Foundation. The talk that I will be giving roughly corresponds to two publications, one at EuroSys 2021 (which also received the best paper award) and one at ICFP 2021."

+

Bio

+

Excerpt from website:

+

"I am a PhD student at the University of Pennsylvania, where I am fortunate to be advised by Rajeev Alur. My research interests lie in the intersection of programming languages, distributed systems, and software verification. I have done an internship in Microsoft Research, where I worked with Sebastian Burckhardt, and an internship in the Automated Reasoning Group in AWS, where I worked with Daniel Schwartz-Narbonne. Before my PhD, I did my undergraduate studies at the National Technical University of Athens, where I was advised by Kostis Sagonas."


+ +
Squashing Bugs and Empowering Programmers with User-Centered Programming Language Design +
+Michael Coblenz +University of Maryland +https://www.cs.umd.edu/~mcoblenz/ +
+10 December, 2021 1:00PM +WVH 366 and Teams +
+

Abstract

+

Programming languages are simultaneously formal systems and user interfaces with which programmers work. Unfortunately, programmers have a hard time writing safe software: serious bugs and security vulnerabilities are common. In many cases, however, languages with strong safety guarantees have been hard to use, and safer languages have seen slow adoption. In this talk, I’ll discuss user-centered design methods I developed to help language designers create languages that are easier to use. I’ll show how I created and evaluated Glacier, an extension for Java that enforces immutability, Obsidian, a new smart contract language that uses a linear type system, and Bronze, a new garbage collector for Rust. In each case, I found that the resulting language helps programmers write software more effectively than with prior approaches.

+

Bio

+

Excerpt from website:

+

"I completed my Ph.D. in the Carnegie Mellon University Computer Science Department. I was a student of Jonathan Aldrich and Brad A. Myers. I also collaborated closely with Joshua Sunshine. Now, I’m a postdoctoral fellow at the University of Maryland. I work with Michael Hicks and Adam Porter."

+

"I spent ten years at Apple. For eight years, I was a full-time software engineer on the iWork team, focusing on Numbers. I worked on versions for macOS, iOS, and iCloud. If you’re still using some other spreadsheet app, give Numbers a try!"


+ +
Beyond Collapsing Towers: Staging a Relational Interpreter +
+Nada Amin +Harvard University +https://namin.seas.harvard.edu/ +
+29 October, 2021 1:00PM +WVH 366 and Zoom +
+

Abstract

+

Staging -- splitting a program into stages, the code generator and the generated code -- can mechanically turn an interpreter into a compiler. Going further, we can collapse a tower of interpreters into a one-pass compiler. +An interpreter written in a relational programming language (miniKanren) for a functional programming language (subset of Racket) can not only evaluate an expression to a value, but also synthesize an expression (or more generally, term) for a hole -- in particular, turning functions into relations. +We adapt staging to the relational setting, turning functions into relations -- and more -- with reduced interpretive overhead, promising a tractable avenue for the synthesis of functional programs.

+

Bio

+

"Nada Amin is an Assistant Professor of Computer Science at Harvard SEAS. Previously, she was a University Lecturer in Programming Languages at the University of Cambridge; a member of the team behind the Scala programming language at the Ecole Polytechnique Federale de Lausanne (EPFL), where she pursued her PhD; and a software engineer at Google, on the compiler infrastructure supporting Gmail and other Google Apps. She holds bachelor of science and master of engineering degrees from the Massachusetts Institute of Technology (MIT)."


+ +
Manifest Deadlock-Freedom for Shared Session Types +
+Stephanie Balzer +Carnegie Mellon University +http://www.cs.cmu.edu/~balzers/ +
+13 November, 2020 12:00PM +Video Conference +
+

Abstract

+

Shared session types generalize the Curry-Howard correspondence between intuitionistic linear logic and the session-typed pi-calculus with adjoint modalities that mediate between linear and shared session types, giving rise to a programming model where shared channels must be used according to an acquire-release discipline. The resulting language recovers the expressiveness of the untyped asynchronous pi-calculus and accommodates practical programming scenarios, such as shared servers, file systems etc. The resulting language is implemented as a prototype in C0 (a safe subset for C used to teach freshmen at CMU) and as a DSL in Rust.

+

The generalization, however, comes at the cost of deadlock-freedom, a property which holds for the purely linear fragment. In this talk, I report on recent work that makes up for this deficiency and introduces a type system of linear and shared session types in which the adjoint modalities are complemented with possible worlds to rule out cyclic dependencies between acquires and synchronizations. I distill the key abstractions to reason about cyclic dependencies and the key invariants enforced by the type system to guarantee deadlock-freedom. Time permitted, I report on ongoing work that also uses possible world modalities for type-based verification of program properties of interest.

+

Bio

+

Stephanie Balzer is a research faculty in the Principles of Programming group in the Computer Science Department at Carnegie Mellon University. Stephanie obtained her PhD from ETH Zurich under the supervision of Thomas R. Gross. In her PhD work, Stephanie developed the sequential relationship-based programming language Rumer and an invariant-based verification technique for Rumer. Stephanie’s current work focuses on the development of type systems and logics for verifying properties of concurrent programs.


+ +
Grow your own language: The YAFL story +
+Darius Blasband +Raincode +https://www.dariusblasband.com +
+17 February, 2020 11:00AM +WVH 366 +
+

Abstract

+

This talk tells the rather uncommon story of the co-evolution of my company, Raincode, with a home-grown programming language, YAFL. The two are intimately intertwined: Raincode would crumble if YAFL was to disappear, and YAFL would have no reason to exist if Raincode was to stop its activities. YAFL is a strongly-typed object-oriented language originally designed and implemented a quarter of century ago. Our story starts with YAFL’s inception, its design, our motivations (including the usual fantasies of world domination) and the technical underpinnings and some key implementation decisions.

+

Our story is rooted in the real-world where pragmatics reign. I will argue that out design choices and implementation decisions have converged to give YAFL its true value, and, that the fruits of growing our own language are well worth the maintenance burden. The talk will argue that YAFL is a non-DSL, and touch on the good ideas as well as the lousy ones. I will discuss how to evolve a language over time by judiciously adding, as well as removing, features. Above all, it is story about people. The language designer, its implementors, its first users, even its customers are the main. Their interplay sheds a light on how languages are created and maintained.

+

Bio

+

Darius Blasband was born in 1965 and has a PhD from the Université Libre de Bruxelles. His focus is legacy code modernization, articulated around compilers for legacy languages. He founded Raincode (https://www.raincode.com), and designed its core technology. Above all, Darius is a geek. Building software systems is what he likes best—after his family, chocolate, and on par with music.


+ +
Automatic Differentiation in PCF +
+Damiano Mazza +CNRS, Université Sorbonne Paris Nord +https://lipn.univ-paris13.fr/~mazza/ +
+28 January, 2020 10:00AM +WVH 366 +
+

Abstract

+

Automatic differentiation (AD) is the science of efficiently computing the derivative (or gradient, or Jacobian) of functions specified by computer programs. It is a fundamental tool in several fields, most notably machine learning, where it is the key for training deep neural networks. Albeit AD techniques natively focus on a restricted class of programs, namely first-order straight-line programs, the rise of so-called differentiable programming in recent years has urged for the need of applying AD to complex programs, endowed with control flow operators and higher-order combinators, such as map and fold. In this talk, I will discuss the extension of AD algorithms to PCF, a(n idealized) purely functional programming language. We will first consider the simply-typed lambda-calculus, showing in particular how linear negation is related to reverse-mode AD (a.k.a., backpropagation), and then see how the extra features of PCF, namely full recursion and conditionals, may be dealt with, stressing the difficulties posed by the latter.

+

Joint work with Aloïs Brunel (Deepomatic) and Michele Pagani (IRIF, Université de Paris).

+

Bio

+

Damiano Mazza is chargé de recherche (researcher) at CNRS, working at the CS lab of Université Sorbonne Paris Nord, a position he has held since 2008. He obtained his Ph.D. in 2006 at the Institut de Mathématiques de Luminy, in Marseille. Before that, he studied CS Engineering in Rome, Italy, which is where he is from. His research interest are at the interface between logic in computer science, the theory of programming languages and complexity theory, with a bias towards the viewpoint provided by linear logic and category theory.


+ +
CertiCoq: Current and Future Directions +
+Zoe Paraskevopoulou +Princeton University +http://zoep.github.io +
+3 December, 2019 10:00AM +WVH 366 +
+

Abstract

+

In this talk, I will give a high-level overview of CertiCoq, a verified compiler from Coq to C, focusing on my research on CertiCoq's backend. The backend of CertiCoq is based on a micro-pass compiler architecture: each stage in its pipeline performs a simple task, yet they combine to generate efficient code. The micro-pass design along with our proof framework based on logical relations greatly facilities modular reasoning and proof reuse. Furthermore, I will show how this logical relation framework can be extended to provide correctness guarantees for programs compiled separately with CertiCoq, using perhaps different sets of optimizations. Lastly, I will discuss future research directions with CertiCoq, such as provably correct interoperation of Coq and C using the Verified Software Toolchain.

+

Bio

+

Zoe Paraskevopoulou is a 5th year PhD student at Princeton University, working with professor Andrew Appel. Prior to that she obtained her master's degree from École normale supérieure Paris-Saclay and her undergraduate degree from National Technical University of Athens. She has done several internships at institutes such as INRIA Paris, Max Planck Institute of Software Systems, Microsoft Research and Facebook.


+ +
Faster, Safer, and Cheaper Serverless Computing using Language-Based Techniques +
+Arjun Guha +University of Massachusetts, Amherst +https://people.cs.umass.edu/~arjun/main/home/ +
+12 November, 2019 10:00AM +WVH 366 +
+

Abstract

+

Serverless computing is a cloud computing abstraction that makes it easier to write robust, large-scale web services. In serverless computing, programmers write what are called serverless functions, and the cloud platform transparently manages the operating system, resource allocation, load-balancing, and fault tolerance. When demand spikes, the platform automatically allocates additional hardware and manages load-balancing; when demand falls, the platform silently deallocates idle resources; and when the platform detects a failure, it transparently retries affected requests.

+

In this talk, we will see how language-based techniques can make serverless computing faster, safer, and cheaper for programmers. First, we present a detailed, operational model of a serverless computing platform, which shows that serverless computing is a leaky abstraction. We then derive a less-leaky "naive semantics" and precisely characterize how the two semantics coincide. Second, we extend the operational semantics with a domain-specific language for composing serverless functions, and show that the language design lends itself to an efficient implementation that lowers the cost of building modular applications. Finally, we present a "serverless function accelerator" that significantly reduces the latency of a large class of functions using language-based sandboxing and speculative optimizations.

+

Bio

+

Arjun Guha is an associate professor of Computer Science at the University of Massachusetts Amherst. Using the tools and techniques of programming languages, his research addresses security, reliability, and performance problems in web applications, systems, networking, and robotics. He received a PhD in Computer Science from Brown University in 2012 and a BA in Computer Science from Grinnell College in 2006.


+ +
Understanding and evolving the Rust programming language +
+Ralf Jung +MPI-SWS +https://people.mpi-sws.org/~jung/ +
+4 October, 2019 11:00AM +WVH 366 +
+

Abstract

+

Rust is a young and actively developed "systems programming language" that promises to overcome the seemingly fundamental tradeoff between high-level safety guarantees and low-level control over resource management. This talk is about my research on understanding and evolving this groundbreaking new language.

+

The first part of the talk is about building formal foundations for the existing Rust language. Rust employs a strong, ownership-based type system to rule out common systems programming anomalies, but then extends the expressive power of this core type system through libraries that internally use unsafe features. Unfortunately, this ``extensible'' aspect of the Rust language makes Rust's safety claims rather non-obvious, and there is good reason to question whether they actually hold. To address this, we have developed RustBelt, the first formal (and machine-checked) safety proof for a language representing a realistic subset of Rust. Our proof supports the extensible nature of Rust's safety story in the sense that, for each new Rust library that uses unsafe features, we can say what verification condition it must satisfy in order for it to be deemed a safe and compositional extension to the language.

+

The second part is about evolving the semantics of Rust to better support compiler optimizations. In particular, Rust's reference types provide strong aliasing guarantees, which a clever optimizing compiler should be able to exploit. However, if no care is taken, unsafe code (which plays an essential role in the Rust ecosystem, as mentioned above) can subvert these guarantees. To address this, we have developed Stacked Borrows, a new operational semantics for memory accesses in Rust. On the one hand, Stacked Borrows defines an aliasing discipline and declares programs violating it to have undefined behavior, thus making it possible for us to formally verify the soundness of a number of useful type-based intraprocedural optimizations. On the other hand, we have also implemented the Stacked Borrows semantics in Miri, an interpreter for Rust, and run large parts of the Rust standard library test suite in Miri. In so doing, we have validated that our semantics is sufficiently permissive to account for real-world unsafe Rust code.

+

Bio

+

Ralf Jung is a PhD student at MPI-SWS in Saarbrücken, where he previously completed his undergraduate degree with Saarland University. Ralf’s research interests center around separation logic, operational semantics for low-level languages, (semantic) type systems, and mechanized+modular proof for program verification. Ralf also loves to learn about anything people do with Rust and the ways in which formal methods could help.


+ +
Towards Gradually Typed Nominal Languages +
+Ross Tate +Cornell University +https://http://www.cs.cornell.edu/~ross/ +
+26 September, 2019 2:00PM +WVH 366 +
+

Abstract

+

Nom demonstrated that efficient and well-behaved gradual typing is possible. But it did so by assuming all code was written using nominal patterns in a simplistic type system. The nominal patterns limit the applicability to existing untyped code, and the simplistic type system limits the applicability to modern industry languages. This talk will present recent results addressing both of these limitations. I will illustrate an implemented design that enables structural patterns to mix with nominal patterns while also providing a variant of the gradual guarantee that ensures interoperability between these patterns and a semantics-preserving migration path from structural patterns into nominal patterns. I will further demonstrate that Nom's design strategy scales to parametric polymorphism, where the key challenge is not actually generic classes or interfaces but rather generic methods and type-argument inference. Although preliminary, experiments suggest that these more expressive designs can still be implemented relatively efficiently, altogether indicating that gradual typing is a viable feature for nominal languages.

+

Bio

+

Ross Tate is an Assistant Professor in Computer Science at Cornell University and the Director of the Single Open Intermediate Language Initiative. His research combines theoretical foundations with studies of programmer behavior to develop language designs that ascribe to both academic and industrial principles. He collaborates with major programming languages such as Kotlin, WebAssembly, Julia, Ceylon, and Java to inform his work from practice and incorporate his work into practice. Due to his research contributions and industry impact, he was awarded the Dahl Nygaard Junior Prize in 2017.


+ +
Asymmetric Numeral Systems +
+Jeremy Gibbons +University of Oxford +http://www.cs.ox.ac.uk/jeremy.gibbons/ +
+19 June, 2019 10:00AM +WVH 366 +
+

Abstract

+

Asymmetric Numeral Systems (ANS) are an entropy-based encoding method introduced by Jarek Duda in 2007, combining the Shannon-optimal compression effectiveness of arithmetic coding with the execution efficiency of Huffman coding. Existing presentations of the ANS encoding and decoding algorithms are somewhat obscured by the lack of suitable presentation techniques; I will present an equational derivation, calculational where it can be, and highlighting the creative leaps where it cannot. This is work to appear at Mathematics of Program Construction in October.

+

Bio

+

Jeremy Gibbons is Professor of Computing at the University of Oxford, and head of the Algebra of Programming research group there. His research interests are in programming languages, especially functional programming. He is Editor-in-Chief of the Journal of Functional Programming, Past Chair of IFIP WG2.1 on Algorithmic Languages and Calculi, and Past Vice Chair of ACM SIGPLAN.


+ +
Safe programming of IoT devices +
+Carlos Mão de Ferro +Faculdade de Ciências da Universidade de Lisboa // UMass Boston +https://github.com/cajomferro/ +
+29 May, 2019 10:00AM +WVH 366 +
+

Abstract

+

The Internet of Things (IoT) has the potential for being the next industrial revolution, envisioning billions of embedded devices. Because of their limited resources, IoT devices present specific challenges to their programming and deployment. For instance, limited battery lifetime imposes extreme energy-savings efforts, and low RAM and CPU processing power not only restrict computation power but can also lead to unexpected runtime errors (e.g, out of memory).

C and C++ languages are often preferred when programming these devices. These languages deliver good performance but are notoriously unsafe and ill-suitable for proving high-level properties specific to HW usage (e.g., sending a message only when Wi-Fi module is connected). Invalid program behaviors can lead to silent errors and energy wasting, resulting in the high-cost of reprogramming thousands of devices.

In this presentation, I will introduce Pekoe, a language for programming IoT devices that can determine, at compile time, the correct usage of hardware components. Pekoe introduces Harel Statecharts as types to model the hardware behavior. The compiler checks every possible execution trace of a program against the hardware model (the type) thus ensuring the correct order of system calls (e.g., sending a message via Wi-Fi, or suspending the CPU).

+

Bio

+

Carlos Mão de Ferro is a visiting Fulbright scholar at UMass Boston (February to July 2019), and is a PhD candidate in Computer Science, University of Lisbon, Portugal.

His research focuses on new programming languages techniques that deliver a safe and resource-aware programming of embedded devices, thus promoting a reliable and sustainable Internet of Things (IoT) environment.

Carlos has been working on WSNs and embedded software for 10 years. After his master thesis conclusion in 2013, he alternated between the academia and the industry worlds. Carlos developed a smart irrigation system for the Green by Web startup that won the Smart Open Lisboa Startup contest in 2017.

In general, Carlos is curious about Science and driven by the desire to understand how things work and how discoveries and inventions come to see the light of day. He enjoys reading topics about Physics, Astronomy and Biology. He is also passionate about music, having learned to play the flute and the piano as a child.

During his stay at UMB, Carlos will be particularly interested in foster collaborations with researchers from IoT-related areas.


+ +
A rehabilitation of message-passing concurrency +
+Frank Pfenning +Carnegie Mellon University +http://www.cs.cmu.edu/~fp/index.html +
+2 May, 2019 10:00AM +WVH 366 +
+

Abstract

+

Recently, there has been a lot of research on shared-memory concurrency. Nevertheless, programmers are discouraged from using it because of the difficulty of writing clear, correct programs. This is embodied for example in the Go Language slogan “Do not communicate by sharing memory; instead, share memory by communicating.” But do we have the right abstractions for message-passing concurrent programming? I argue that we do not (yet!) because concurrency constructs are usually bolted on to an existing language with an entirely different semantic foundation. I will review some recent progress on designing better abstractions based on strong logical and type-theoretic principles. Multiple themes from functional and object-oriented programming will re-emerge from these foundations in a new form, including (ha!) shared memory.

+

Bio

+

Frank Pfenning studied Mathematics and Computer Science at the Technical University Darmstadt and then left for Carnegie Mellon University on a Fulbright scholarship where he obtained his Ph.D. in Mathematics in 1987 under the supervision of Professor Peter Andrews. He subsequently joined the Department of Computer Science at Carnegie Mellon University as research faculty where he became Professor in 2002 and served as Director of Graduate Programs from 2004 to 2008 and Associate Dean for Graduate Education from 2009 to 2010. He was the Joseph F. Traub Professor of Computer Science and Head of the Computer Science Department from 2013 to 2018. He has spent time as visiting scientist at the Max-Planck-Institute for Computer Science in Saarbrücken, as Alexander-von-Humboldt fellow at the Technical University Darmstadt, and as visiting professor at École Polytechnique and INRIA-Futurs. He has advised 24 completed Ph.D. theses and won the Herbert A. Simon Award for Teaching Excellence in the School of Computer Science in 2002. He served as trustee, vice president, and president of CADE, Inc., the governing body of the International Conference on Automated Deduction, and on advisory boards for INRIA, the Max-Planck-Institute for Computer Science, and Seoul National University. He has chaired several conferences and program committees, including CADE and LICS, and has been a member of the editorial boards for Theoretical Computer Science, Journal of Automated Reasoning, and the Journal of Symbolic Computation. He was named Fellow of the ACM in 2015. His research interests include programming languages, logic and type theory, logical frameworks, automated deduction, and computer security. In his spare time he enjoys playing squash, running, hiking, cooking, and reading.


+ +
Optimizing the Automated Programming Stack +
+James Bornholt +University of Washington +https://homes.cs.washington.edu/~bornholt/ +
+27 March, 2019 10:00AM +WVH 366 +
+

Abstract

+

The scale and pervasiveness of modern software poses a challenge for programmers: software reliability is more important than ever, but the complexity of computer systems continues to grow. Automated programming tools are powerful weapons for programmers to tackle this challenge: verifiers that check software correctness, and synthesizers that generate new correct-by-construction programs. These tools are most effective when they apply domain-specific optimizations, but doing so today requires considerable formal methods expertise. In this talk, I present a new application-driven approach to optimizing the automated programming stack underpinning modern domain-specific tools. I will demonstrate the importance of programming tools in the context of memory consistency models, which define the behavior of multiprocessor CPUs and whose subtleties often elude even experts. Our new tool, MemSynth, automatically synthesizes formal descriptions of memory consistency models from examples of CPU behavior. We have used MemSynth to synthesize descriptions of the x86 and PowerPC memory models, each of which previously required person-years of effort to describe by hand, and found several ambiguities and underspecifications in both architectures. I will then present symbolic profiling, a new technique we designed and implemented to help people identify the scalability bottlenecks in automated programming tools. These tools use symbolic evaluation, which evaluates all paths through a program, and is an execution model that defies both human intuition and standard profiling techniques. Symbolic profiling diagnoses scalability bottlenecks using a novel performance model for symbolic evaluation that accounts for all-paths execution. We have used symbolic profiling to find and fix performance issues in 8 state-of-the-art automated tools, improving their scalability by orders of magnitude, and our techniques have been adopted in industry. Finally, I will give a sense of the importance of future application-driven optimizations to the automated programming stack, with applications that inspire improvements to the stack and in turn beget even more powerful automated tools.

+

Bio

+

James Bornholt is a Ph.D. candidate in the Paul G. Allen School of Computer Science & Engineering at the University of Washington, advised by Emina Torlak, Dan Grossman, and Luis Ceze. His research interests are in programming languages and formal methods, with a focus on automated program verification and synthesis. His work has received an ACM SIGPLAN Research Highlight, two IEEE Micro Top Picks selections, an OSDI best paper award, and a Facebook Ph.D. fellowship.


+ +
Mechanized Verification of Graph-manipulating Programs +
+Aquinas Hobor +Yale-NUS College and the School of Computing, National University of Singapore +https://www.comp.nus.edu.sg/~hobor/ +
+22 March, 2019 10:00AM +WVH 366 +
+

Abstract

+

Graph-manipulating programs such as garbage collectors are notoriously unpleasant to reason about, which is unfortunate since such programs are also used in critical algorithms and systems. We show how to verify such programs written in real C in a fully machine-checked context.

+

Bio

+

Aquinas Hobor is an Assistant Professor with a joint appointment at Yale-NUS College and the School of Computing, National University of Singapore. From 2008-2011 he was a Lee Kuan Yew Postdoctoral Fellow after getting his PhD from Princeton University. He grew up near Chicago and his Go ranking is approximately 1-dan.


+ +
From C to Java EE to Node.js: A Journey in Industrial Program Analysis +
+François Gauthier +Oracle Labs Australia +https://labs.oracle.com/pls/apex/f?p=labs:bio:0:2080 +
+19 March, 2019 10:00AM +WVH 462 +
+

Abstract

+

I will divide my presentation in two short talks. In the first part of my presentation, I will describe how static analysis of C/C++, our lab's initial expertise, widely differs from static analysis of Java EE, and describe some of the challenges we encountered in the Wafer project. In the second part of my talk, I will describe how we scale dynamic security analysis for Node.js with taint inference, and how the Affogato project compares to state-of-the-art.

+

Bio

+

François Gauthier graduated in 2014 from Polytechnique Montreal with a PhD in Software Engineering. The same year, he joined the program analysis team at Oracle Labs Australia, under the direction of Dr. Cristina Cifuentes, to start and lead the Wafer project for static vulnerability detection in Java EE. In June 2017, he transitioned to, and is now leading the Affogato project for dynamic security analysis of Node.js. Apart from application security, François is also exploring program analysis and machine learning approaches to detect document-based malware as well as fuzzing techniques to automatically generate test inputs.


+ +
+

Abstract

+

Constant-time programming is an established discipline to secure programs against timing attackers. Several real-world secure C libraries such as NaCl, mbedTLS, or Open Quantum Safe, follow this discipline. We propose two advanced compile-time verification techniques to enforce this discipline. The first one operates at assembly level with help from alias informations that are computed at source level and transmitted by the compiler. The second one operates at source level, using advanced abstract interpretation techniques. For this last approach, we also study how to ensure that the compiler will not break the property during compilation. These techniques have been implemented in the CompCert compiler toolchain and tested on several crypto libraries. These works have been presented at CCS’14, CSF’17 and ESORICS’17.

+

Bio

+

I am professor of Computer Science and head of the Department of Computer Science at ENS Rennes. My research activities take place in the Celtique team (joint team between University of Rennes, ENS Rennesand Inria Rennes, under the IRISA joint lab). I received a Ph.D. in Computer Science from the University of Rennes, France, in 2005, and a Habilitation à diriger les recherches in Computer Science from the ENS Cachan, France, in 2012. I joined ENS Cachan in September 2013 as full professor (in 2014, the brittany extension of ENS Cachan has becomed ENS Rennes). Between 2007 and 2013, I was a full research at INRIA Rennes research center. Earlier, I was holding a postdoc position at INRIA Sophia-Antipolis under the supervision of Gilles Barthe. In the 2011-13 academic years, I took a sabbatical and visited Jan Vitek's group at Purdue University, Indiana, USA, during the first year, and then Greg Morrisett's group at Harvard University, Cambridge, USA, during the second year. My research interests include formal methods, programming languages, program verification, software, and system security. I am a long time happy user of the Coq proof assistant and the theory of Abstract interpretation. More recently I have been conducting several researches about the verified C compiler CompCert. My research is currently funded by an ERC Consolidator Grant "VESTA: VErified STatic Analysis platform" (2018-2023).


+ +
Compare Less, Defer More: Scaling Value-Contexts Based Whole-Program Heap Analyses +
+Manas Thakur +PACE Lab, Indian Institute of Technology Madras +http://www.cse.iitm.ac.in/~manas/ +
+21 February, 2019 10:00AM +Ryder 431 +
+

Abstract

+

The precision of heap analyses determines the precision of several associated optimizations, and has been a prominent area in compiler research. It has been shown that context-sensitive heap analyses are more precise than the insensitive ones, but their scalability continues to be a cause of concern. Though the value-contexts approach improves the scalability of classical call-string based context-sensitive analyses, it still does not scale well for several popular whole-program heap analyses. In this talk, I will discuss our three-staged approach that lets us scale complex whole-program value-contexts based heap analyses for large programs. The ideas involved are based on an important observation that we do not need to compare the complete value-contexts at each call-site. Apart from this, I will also give an overview of another work, named PYE (for "precise-yet-efficient”). PYE helps obtain precise results during just-in-time compilation efficiently, by splitting the work between the static Java compiler and the C2 compiler of the HotSpot JVM.

+

Bio

+

Manas is a PhD student under V. Krishna Nandivada at PACE Lab, IIT Madras. His current research focuses on balancing the precision-scalability tradeoffs while analyzing programs written in object-oriented languages. He has worked with the Soot optimization framework for static analyses, and the HotSpot JVM for just-in-time analyses. Apart from work, he writes articles on GitHub and Medium (technical and otherwise!), and experiments with Vim plugins when feeling geeky.


+ +
Promising-ARM/RISC-V: a simpler and faster operational concurrency model +
+Chung-Kil Hur +Software Foundations Lab, Seoul National University +https://sf.snu.ac.kr/gil.hur/ +
+15 February, 2019 10:00AM +Ryder 431 +
+

Abstract

+

For ARMv8 and RISC-V, there are concurrency models in two styles, extensionally equivalent: axiomatic models, expressing the concurrency semantics in terms of global properties of complete executions; and operational models, that compute incrementally. The latter are in an abstract micro-architectural style: they execute each instruction in multiple steps, out-of-order and with explicit branch speculation. This similarity to hardware implementations has been important in developing the models and in establishing confidence, but involves complexity that, for programming and model-checking, one would prefer to avoid. In this talk, I will present new more abstract operational models for ARMv8 and RISC-V, and an exploration tool based on them. The models compute the allowed concurrency behaviours incrementally based on thread-local conditions and are significantly simpler than the existing operational models: executing instructions in a single step and (with the exception of early writes) in program order, and without branch speculation. We prove the models equivalent to the existing ARMv8 and RISC-V axiomatic models in Coq. The exploration tool is the first such tool for ARMv8 and RISC-V fast enough for exhaustively checking the concurrency behaviour of a number of interesting examples. We demonstrate using the tool for checking several standard concurrent datastructure and lock implementations such as Michael-Scott queue and Chase-Lev deque, compiled from C++ and Rust with GCC or Clang -O3, and for interactively stepping through model-allowed executions for debugging.

+

Bio

+

Chung-Kil Hur is an associate professor in Department of Computer Science and Engineering at Seoul National University. Previously he worked as a postdoctoral researcher at Microsoft Research Cambridge,Max Planck Institute for Software Systems (MPI-SWS) and Laboratoire PPS. He obtained a Ph.D. from University of Cambridge and a B.Sc. in both Computer Science and Mathematics from Korea Advanced Institute of Science and Technology (KAIST). His current research interests are in semantics of low-level languages such as C,Rust,LLVM IR, relaxed memory concurrency, formal verification of software such as compiler and OS, and interactive theorem provers such as Coq.


+ +
Path-Based Function Embedding and Its Application to Error-Handling Specification Mining +
+Cindy Rubio González +University of California, Davis +http://web.cs.ucdavis.edu/~rubio/ +
+14 February, 2019 10:00AM +Ryder 431 +
+

Abstract

+

Identifying relationships among program elements is useful for program understanding, debugging, and analysis. One such kind of relationship is synonymy. Function synonyms are functions that play a similar role in code; examples include functions that perform initialization for different device drivers, and functions that implement different symmetric-key encryption schemes. Function synonyms are not necessarily semantically equivalent and can be syntactically dissimilar; consequently, approaches for identifying code clones or functional equivalence cannot be used to identify them. In this talk I will present our recent work Func2vec, a technique that learns an embedding that maps each function to a vector in a continuous vector space such that vectors for function synonyms are in close proximity. We compute the function embedding by training a neural network on sentences generated using random walks over the interprocedural control-flow graph. We show the effectiveness of Func2vec at identifying function synonyms in the Linux kernel, and its applicability to the problem of mining error-handling specifications in Linux file systems and drivers.

+

Bio

+

Cindy Rubio-Gonzalez is an Assistant Professor of Computer Science at the University of California, Davis. Prior to that position, she was a Postdoctoral Researcher in the EECS Department at the University of California, Berkeley. She received her Ph.D. in Computer Science from the University of Wisconsin--Madison in 2012. Cindy's work spans the areas of Programming Languages and Software Engineering, with a focus on program analysis for automated bug finding, program optimization,and software reproducibility. She is particularly interested in the reliability and performance of systems software and scientific computing applications. She currently leads the BugSwarm project,which collects and automatically reproduces thousands of real-world bugs from public software repositories. Among other awards, Cindy is a recipient of an NSF CAREER award 2018, Hellman Fellowship 2017, and UC Davis CAMPOS Faculty Award 2014.


+ +
What Spectre means for language implementors +
+Ben L. Titzer +Google +http://research.google.com/pubs/BenTitzer.html +
+13 February, 2019 10:00AM +Ryder 155 +
+

Abstract

+

Until now, CPU designers and computer scientists have assumed Vegas rules at the hardware level: what happens in speculation stays in speculation. Yet in the wake of the Spectre and Meltdown attacks, it has become clear a new, massive class of security vulnerabilities awaits us at the microarchitectural level because this assumption is simply false. As language designers and implementors familiar with building towers of abstractions, we have assumed that virtualization through emulation made the worlds below the Turing machine undetectable, hidden behind a mathematically perfect mirror. This talk will explore how we have now learned to see through that mirror, into the very bizarre and alien world of microarchitectures, illuminating a tiny world of astounding complexity that holds profound implications for language security.

+

Bio

+

Ben L. Titzer leads Google's WebAssembly team in Munich. Before starting the WebAssembly project with Luke Wagner from Mozilla, he led the team that built the TurboFan optimizing compiler which now powers JavaScript and WebAssembly in V8. He graduated with a BS from Purdue University in 2002 and MS and PhD from UCLA in 2004 and 2007. His interests include safe programming languages for systems programming, compilers, virtual machines, nature and playing guitar.


+ +
Whip: Higher-order Contracts for Modern Services +
+Lucas Waye +Facebook +http://lucaswaye.com/ +
+6 February, 2019 10:00AM +WVH 366 +
+

Abstract

+

Modern service-oriented applications forgo semantically rich protocols and middleware when composing services. Instead, they embrace the loosely-coupled development and deployment of services that communicate via simple network protocols. Even though these applications do expose interfaces that are higher-order in spirit, the simplicity of the network protocols forces them to rely on brittle low-level encodings. To bridge the apparent semantic gap, programmers introduce ad-hoc and error-prone defensive code. Inspired by Design by Contract, we choose a different route to bridge this gap. We introduce Whip, a contract system for modern services. Whip (i) provides programmers with a higher-order contract language tailored to the needs of modern services; and (ii) monitors services at run time to detect services that do not live up to their advertised interfaces. Contract monitoring is local to a service. Services are treated as black boxes, allowing heterogeneous implementation languages without modification to services' code. Thus, Whip does not disturb the loosely coupled nature of modern services.

+

Bio

+

I'm currently a Software Engineer at Facebook thinking about privacy in the data warehouse. Previously I helped build TiVo's Targeted Audience Delivery platform. And before that I worked on the backend for Timeful (acquired by Google) and on Hulu's analytics platform. In graduate school I worked on Whip, a contract monitor for microservices. I received a Ph.D. and Master's degree from Harvard under the guidance of Stephen Chong and a Bachelor's degree from Cornell.


+ +
Better Stories, Better Languages — Reframing from Programs to Programming +
+François-René Rideau +Alacris +http://fare.tunes.org +
+30 January, 2019 10:00AM +WVH 366 +
+

Abstract

+

Software tools imply a story. New stories can help invent new tools. Explicit stories are a great meta-tool... and provide a systematic way to improve the design of programming language design. To illustrate my point, I will present a series of pairs of stories, each time approaching a same topic with very different consequences. Many of my stories will be familiar to most of you, some other stories less so. Hopefully they will be enlightening as well as entertaining... and so will the meta-story be. (Earlier versions of this talk were given at International Lisp Conference 2009, Lisp NYC 2014-10, and more recently LambdaConf 2017.)

+

Bio

+

François-René Rideau is Co-Founder and Chief Architect at Alacris.io,a company developing a "Blockchain Layer 2 Operating System" using formal methods. Once founder of the TUNES Project, he left academic research on Programming Languages and Distributed Systems to build industrial software at ITA Software, then Google and Bridgewater Associates, before he finally became his own Entrepreneur. Long-standing member of the Lisp community, he runs the Boston Lisp Meeting and still co-maintains ASDF (the Common Lisp build system),though he recently jumped ship to Gerbil Scheme, and now uses OCaml at work.


+ +
Goblins and Spritely: from the actor model to distributed virtual worlds +
+Christoper Lemmer Webber +ActivityPub +https://dustycloud.org/ +
+5 December, 2018 10:00AM +WVH 366 +| Notes
+

Abstract

+

Goblins is an actor model implementation for Racket, inspired by object-capability secure distributed progamming languages such as E with a few new twists of its own thrown in. Goblins is the foundation for a project to build secure, player/user-programmable virtual worlds named Spritely. Spritely builds on the success of the ideas from the ActivityPub protocol, which has succeeded at connecting together various distributed social network projects, extending them to pull in rich interactive and game aspects. See what is built so far, what's coming next, and how the actor model and object capabilities tie together all these ideas to enable a robust and secure distributed system.

+

Bio

+

Christopher Lemmer Webber is a user freedom advocate who works on distributed social network technology. They are most well known for their work on the W3C ActivityPub federated protocol, which connects over 1.5 million users across the distributed social web. Chris enjoys programming in lisp/scheme flavored languages, especially Racket.


+ +
+

Abstract

+

Programming languages differ a lot in the ways of their development. Who is responsible for including new features and removing obsolete ones? What is a process of doing that? Is there any more or less formal language definition? How respected is that definition by the compilers? How is this changed if there is only one compiler? These questions are answered quite differently for various programming languages. Haskell takes its own way which was crystallized relatively recently. We have GHC Proposals in the form of the GitHub pull requests to suggest new language features. We also have the GHC Steering Committee with its own process to either accept or reject proposals. Finally, there is the Haskell 2020 Language (Haskell Prime) Committee with its own problem of being effectively dead. In this talk, I’d like to describe how this system works for Haskell and why it doesn’t work sometimes. I will discuss the most recent examples of transforming Haskell kinds system towards dependent types (started before the GHC Steering Committee was formed) and introducing linear types in Haskell (subject to the full-blown Committee discussion) among others. I will also characterize the current state and perspectives of the Haskell standardization process.

+

Bio

+

Vitaly Bragilevsky serves as both the Haskell 2020 Language Committee and the GHC Steering Committee member. He works as a Senior Lecturer at the Southern Federal University in Rostov-on-Don, Russia where he teaches undergraduate students functional programming and theory of computations. He is currently a grantee of the Fulbright Faculty Development Program holding temporary Courtesy Research Assistant position at the University of Oregon under the supervision of Prof. Zena Ariola. Vitaly Bragilevsky translated into Russian and edited translations of the several books on Haskell and the theory of programming languages. He is the author of ‘Haskell in Depth’ (Manning Publications, available via Manning’s early access program).


+ +
Synthesizing Solutions to the Expression Problem +
+George T. Heineman +Worcester Polytechnic Institute +https://www.wpi.edu/people/faculty/heineman +
+17 October, 2018 10:00AM +WVH 366 +
+

Abstract

+

The Expression Problem (EP) describes a common software issue that appears regardless of programming language or programming paradigm. As coined by Philip Wadler in 1998, EP is a new name for an old problem. Given a data type defined by cases, consider two independent extensions, namely, adding new cases to the data type and new functions over the datatype. Dozens of papers in the research literature present their solutions to the EP, using a variety of programming languages and approaches. In this talk I present a Scala-based framework that can synthesize full solutions to the EP problem. The first step is to model the domain of interest (such as mathematical expressions or geometric shapes) to ensure that our synthesized code can work with any desired application domain. Second, we demonstrate how to synthesize code that reproduces a dozen EP solutions from the literature, in C++, Java and Haskell. We designed a common API for all approaches, independent of programming language. While we do not introduce any new approach for EP, we explore the challenges that EP solutions face with binary methods and producer methods. An important contribution of this work is the scientific reproduction of known solutions from the literature. +This work is based on an ongoing collaboration with with Jan Bessai (TU Dortmund) and Boris Düdder (Univ. Copenhagen).

+

Bio

+

George T. Heineman is an Associate Professor of Computer Science at WPI. His research interests are in Software Engineering, specifically synthesizing software systems from composable units. He is co-author of "Algorithms in a Nutshell (2ed)," published by O'Reilly Media in 2016. Aside from his professional pursuits, George is an avid puzzler. He invented Sujiken(R), a Sudoku variation played on a right-triangle arrangement of cells in which numbers cannot repeat in a horizontal row, vertical column or diagonal in any direction.


+ +
Rehabilitating the POSIX shell +
+Michael Greenberg +Pomona College +http://www.cs.pomona.edu/~michael/ +
+10 October, 2018 10:00AM +WVH 366 +
+

Abstract

+

We build intricate systems with complex algorithms and invariants, aiming for guarantees of correctness and performance... and then we maintain and deploy these systems with shell scripts! What *are* shell scripts? If the POSIX shell is a programming language, what are its syntax and semantics? Can we apply PL tools to reason about the shell? Why haven't prior PL attempts at understanding the shell redeemed it?

+

Bio

+

Michael Greenberg is an assistant professor at Pomona College. He received his BA in Computer Science and Egyptology from Brown University (2007) and his PhD in Computer Science from the University of Pennsylvania (2013). His work has ranged from functional-reactive JavaScript (with Shriram Krishnamurthi at Brown) to runtime verification of higher-order programs using contracts (with Benjamin Pierce at Penn) to software-defined networking (with Dave Walker at Princeton) to present activities focused on Kleene algebra with tests and the POSIX shell. He is always looking for new climbing partners.


+ +
+

Abstract

+

The Parfait static code analysis tool focuses on detecting vulnerabilities that matter in C, C++, Java and PL/SQL languages. Its focus has been on key items expected out of a commercial tool that lives in a commercial organization, namely, precision of results (i.e., high true positive rate), scalability (i.e., being able to run quickly over millions of lines of code), incremental analysis (i.e., being able to run over deltas of the code quickly), and usability (i.e., ease of integration into standard build processes, reporting of traces to the vulnerable location, etc). Today, Parfait is used by thousands of developers at Oracle worldwide on a day-to-day basis. In this presentation we’ll sample a flavour of Parfait — we explore some real world challenges faced in the creation of a robust vulnerability detection tool, look into two examples of vulnerabilities that severely affected the Java platform in 2012/2013 and most machines in 2017/2018, and conclude by recounting what matters to developers for integration into today’s continuous integration and continuous delivery (CI/CD) pipelines.

+

Bio

+

Cristina is the Director of Oracle Labs Australia and an Architect at Oracle. Headquartered in Brisbane, the Lab focuses on Program Analysis as it applies to finding vulnerabilities in software and enhancing the productivity of developers worldwide. Prior to founding Oracle Labs Australia, Cristina was the Principal Investigator of the Parfait bug tracking project at Sun Microsystems, then Oracle.


+ +
On the expressive power of user-defined effects +
+Ohad Kammar +University of Oxford +https://www.cs.ox.ac.uk/people/ohad.kammar/main.html +
+3 October, 2018 10:00AM +WVH 366 +
+

Abstract

+

Computational effects, such as mutable state, memory allocation, non-determinism, and I/O interaction, allow programs to implicitly exhibit complex behaviour. I will discuss three abstractions that allow programmers to extend a language with user-defined effects. Control operators (call-cc, shift/reset, etc.) are a well-established abstraction for user-defined effects, and I will consider the specific shift/dollar without answer-type-modification. Since the 90s, functional languages have used monads for user-defined effects, and I will consider Filinski's monadic reflection. In the last decade, the community started looking into Plotkin and Pretnar's handlers for algebraic effects (extensible effects in Haskell, OCaml multicore, the Eff, Frank, Koka, and Links programming languages) as a programming abstraction, and I will consider deep handlers without forwarding. I will compare the relative expressive power of these three abstractions using Felleisen's notion of macro-translations. This comparison demonstrates the sensitivity of relative expressiveness of user-defined effects to seemingly orthogonal language features. This talk is based on the paper: On the expressive power of user-defined effects: effect handlers,monadic reflection, delimited control. Yannick Forster, Ohad Kammar, Sam Lindley, and Matija Pretnar. Proceedings of the 22nd ACM SIGPLAN International Conference on Functional Programming, PACMPL 1(ICFP): 13:1-13:29 (2017),arXiv:1610.09161, DOI: 10.1145/3110257.

+

Bio

+

Ohad Kammar is a postdoctoral research associate at the University of Oxford Department of Computer Science, a Career Development Fellow at Balliol College Oxford, and an incoming Royal Society University Research Fellow at the University of Edinburgh School of Informatics. His fields of interests include programming language modelling and design, logic, and the foundations of computer science.


+ +
From trash to treasure: Timing-sensitive garbage collection +
+Mathias Pedersen +Aarhus University +http://cs.au.dk/~mvp/ +
+19 September, 2018 10:00AM +WVH 366 +
+

Abstract

+

We study information flows arising from timing channels in the presence of automatic memory management. We construct a series of example attacks that illustrate how garbage collectors form a shared resource that can be used to reliably leak sensitive information at a rate of up to 1 byte/sec on a contemporary general-purpose computer. The created channel is also observable across a network connection in a datacenter-like setting. We subsequently present a design of an automatic memory management system and prove that the presented attacks are impossible on garbage collectors satisfying this design. The guarantees provided by the language has been mechanized in the Coq proof assistant.

+

Bio

+

Mathias is a third year PhD student in the Logic and Semantics group at Aarhus University in Denmark, advised by Aslan Askarov. His work is in the area of language-based security, with a focus on provable mitigation of side channels. In general, anything related to compilers and the semantics of programming languages will be on his list of interests.


+ +
Inferring Type Rules for Syntactic Sugar +
+Justin Pombrio +Brown University +http://justinpombrio.net +
+7 August, 2018 11:30AM +WVH 366 +
+

Abstract

+

Type systems and syntactic sugar are both valuable to programmers, but sometimes at odds. While sugar is a valuable mechanism for implementing realistic languages, the expansion process obscures program source structure. As a result, type errors can reference terms the programmers did not write (and even constructs they do not know), baffling them. The language developer must also manually construct type rules for the sugars, to give a typed account of the surface language. We address these problems by presenting a process for automatically reconstructing type rules for the surface language using rules for the core. We have implemented this theory, and show several interesting case studies.

+

Bio

+

Justin Pombrio is a recent PhD graduate from Brown University. His research is mainly in programming languages, with a focus on syntactic sugar, but also includes CS education.


+ +
Debugging Debug Information and Beyond +
+Francesco Zappa Nardelli +Inria Paris – Rocquencourt +http://www.di.ens.fr/~zappa/ +
+15 June, 2018 10:00AM +WVH 366 +
+

Abstract

+

The spectacular results achieved by computer science in the recent years rely on hidden, obscure, and badly specified components that lurk at the very heart of our computing infrastructure. Consider DWARF debug information. Debug information is relied upon by debuggers and plays a key role in the in the implementation of program analysis tools. More surprisingly, debug information can be relied upon by the runtime of high-level programming languages, for instance by C++ to unwind the stack and implement exceptions. The debug information itself can be pervaded by subtle bugs, making the whole infrastructure unreliable. In this talk I will describe techniques and tools to perform validation and synthesis of the DWARF stack unwinding tables. I will also report on adventurous projects that we might build on top of reliable DWARF information.

+

Bio

+

Francesco Zappa Nardelli is a Researcher at Inria Paris – Rocquencourt. His research interests focus on concurrent computation on top of relaxed memory models, ranging from hardware models of modern architectures to high-level programming language specification and compilation. He received his Ph.D. from Université Paris 7 in 2003. Since then, he has worked on language design for distributed computation, type systems for integrating typed and untyped code in scripting languages, and tool support for semantics (including the Ott tool).


+ +
Call-By-Name Gradual Type Theory +
+Max New +Northeastern University +http://maxsnew.github.io/ +
+27 April, 2018 11:30AM +WVH 366 +
+

Abstract

+

I will present Call-by-name (CBN) Gradual Type Theory [1], an axiomatic semantics of CBN gradual typing that directly axiomatizes the extensionality (eta) principles of types and the "gradual guarantee" of [2]. By axiomatizing the "term precision" relation of [2] we can provide a specification for casts as meets and joins with respect to precision. Using our type theory, we then show that the classic function contract ([3]) is in fact the unique implementation of function casts that satisfies extensionality and the gradual guarantee. This shows that the combination of these properties is a very strong constraint on the design of a gradually typed language and any implementation that departs must sacrifice one of these properties (typically extensionality).

[1] Call-by-Name Gradual Type Theory, New and Licata, to appear FSCD '18,https://arxiv.org/abs/1802.00061

[2] Refined Criteria for Gradual Typing, Siek, Vitousek, Cimini and Boyland, SNAPL '15, http://drops.dagstuhl.de/opus/volltexte/2015/5031/

[3] Contracts for Higher-Order Functions, Findler and Felleisen, ICFP '02,https://dl.acm.org/citation.cfm?id=581484

+

Bio

+

Max New is a PhD student at Northeastern University working on the semantic foundations of programming languages. He hates proving the same theorem twice and you should too.


+ +
Compilation of Simple Probabilistic Programs to Gibbs Sampling. +
+John Tristan +Oracle Labs +https://jtristan.github.io/ +
+10 April, 2018 11:45AM +WVH 366 +
+

Abstract

+

One of the several interesting challenges of probabilistic programming is that of compiling probabilistic programs to inference algorithms. One of these inference algorithms, Gibbs sampling, is particularly relevant because it is often statistically efficient, but unfortunately, it is difficult to derive and therefore compile to.

In this talk, after a brief explanation of probabilistic programming and why its relevance to data science, I will explain some of the ideas behind the design of a compiler from (very) simple probabilistic programs to Gibbs sampling. I will also attempt to explain what it would mean for such a compiler to be correct.

+

Bio

+

Jean-Baptiste Tristan is a researcher in the machine learning group at Oracle Labs. He holds a Ph.D. in Computer Science from the French Institute for Research in Computer Science and Automation (INRIA) and a M.Sc. in Computer Science from the Ecole Normale Superieure of Paris.


+ +
Design by Introspection using the D Language +
+Andrei Alexandrescu +D Language Foundation +http://erdani.com +
+6 February, 2018 11:45AM +WVH 366 +
+

Abstract

+

Years ago, D started modestly as an improved offering in the realm of systems programming languages, sharing a good deal of philosophy with C and C++. With time, however, D became a very distinct language with unique features (and, of course, its own mistakes).

One angle of particular interest has been D's ability to perform compile-time introspection. Artifacts in a D program can be "looked at" during compilation. Coupled with a full-featured compile-time evaluation engine and with an ability to generate arbitrary code during compilation, this has led to a number of interesting applications.

This talk shares early experience with using these features of the D language. Design by Introspection is a proposed programming paradigm that assembles designs by performing introspection on components during compilation.

+

Bio

+

Andrei Alexandrescu is a researcher, software engineer, and author. He wrote three best-selling books on programming (Modern C++ Design, C++ Coding Standards, and The D Programming Language) and numerous articles and papers on wide-ranging topics from programming to language design to Machine Learning to Natural Language Processing. Andrei holds a PhD in Computer Science from the University of Washington and a BSc in Electrical Engineering from University "Politehnica" Bucharest. He currently works on the D Language Foundation. http://erdani.com


+ +
Solver-Aided Declarative Programming +
+Molham Aref +Relational AI +# +
+14 December, 2017 10:00AM +WVH 366 +
+

Abstract

+

I will summarize our work on a declarative programming language that offers native language support for model (or instance) finding. This capability can be used to express predictive (e.g. machine learning) and prescriptive (e.g. combinatorial optimization) analytics. The presentation gives an overview of the platform and the language. In particular, it focuses on the important role of integrity constraints,which are used not only for maintaining data integrity, but also, for the formal specification of complex optimization problems and probabilistic programming.

+

Bio

+

Mr. Molham Aref is the Chief Executive Officer of Relational AI. Mr. Aref has more than 25 years of experience in developing and implementing enterprise-grade analytic, predictive, optimization and simulation solutions for the demand chain, supply chain, and revenue management across various industries. Relational AI combines the latest advances in Artificial Intelligence with a understanding of business processes to develop solutions that shape better decisions, improve agility, and reduce risk. Prior to Relational AI, he was co-founder and CEO of LogicBlox where he led the company from inception through a successful sale to Infor. Previously, he was CEO of Optimi (acquired by Ericsson),a leader in wireless network simulation and optimization, and co-founder of Brickstream (renamed Nomi and then acquired by FLIR), a leading provider of computer-vision-based behavior intelligence solutions.


+ +
Compositional Creativity +
+Chris Martens +North Carolina State University +https://sites.google.com/ncsu.edu/cmartens +
+11 December, 2017 11:00AM +WVH 366 +
+

Abstract

+

Creativity is seen as a core cognitive ability for tasks like storytelling, conversation, and aesthetic design. Simulating creative processes with computers enables them to amplify humans' natural creativity and to support applications like interactive narrative, chatbots, and procedural level generators for games. By now, a large number of techniques exist for simulating creativity, such as search, planning, production grammars, logic programming, stochastic modeling from data, and hybrid approaches. However, even the simplest methods are commonly considered programming tasks for "AI experts." High-level programming languages have the potential to broaden participation in authorship of these systems, but only if they are designed with compositionality -- the mathematical principle that's also functional programming's secret weapon -- in mind. This talk will discuss what compositionality has achieved so far for creative computing and what we can do with it next.

+

Bio

+

Chris Martens is an assistant professor in the Computer Science (CSC) Department at NC State, where she is affiliated with the Digital Games research group. Her interests span artificial intelligence, programming languages, and formal methods.


+ +
Efficient Nominal Gradual Typing +
+Fabian Muehlboeck +Cornell University +http://www.cs.cornell.edu/~fabianm/ +
+5 December, 2017 2:00PM +WVH 366 +
+

Abstract

+

Gradual Typing is the idea that we can support both static and dynamic type checking in different parts of the same program in the same language. Most existing work on gradual typing aims to add gradual typing to an existing language. However, this severely constrains some key decisions in creating gradually typed languages, often leading designers to trade off either soundness or efficiency.

+

I will talk about designing a nominal object-oriented language with gradual typing as a core component from the start. This affects key design decisions for the other type system features, as well as the design of the runtime. Experiments with a prototype language I implemented suggest that this is a viable approach to achieve sound, yet efficient gradual typing. I will present those results and talk about some key design challenges that have yet to be solved.

+

Bio

+

Fabian Muehlboeck is a Ph.D. student working with Ross Tate at Cornell University.


+ +
What does CPS have to do with deep learning? +
+Jeffrey M. Siskind +Purdue University +https://engineering.purdue.edu/~qobi/ +
+11 September, 2017 1:30PM +WVH 366 +
+

Abstract

+

Deep learning is formulated around backpropagation, a method for computing gradients of a particular class of data flow graphs to train models to minimize error by gradient descent. Backpropagation is a special case of a more general method known as reverse mode automatic differentiation (AD) for computing gradients of functions expressed as programs. Reverse mode AD imposes only a small constant-factor overhead in operation count over the original computation, but has storage requirements that grow, in the worst case, in proportion to the time consumed by the original computation. This storage blowup can be ameliorated by check pointing, a process that reorders application of classical reverse-mode AD over an execution interval to tradeoff space vs. time. Application of check pointing in a divide-and-conquer fashion to strategically chosen nested execution intervals can break classical reverse-mode AD into stages which can reduce the worst-case growth in storage from linear to logarithmic. Doing this has been fully automated only for computations of particularly simple form, with checkpoints spanning execution intervals resulting from a limited set of program constructs. We show how the technique can be automated for arbitrary computations. Doing so relies on implementing general purpose mechanisms for counting the number of instructions executed by a program, interrupting the execution after a specified number of steps, and resuming the execution with a nonstandard interpretation. We implement these general purpose mechanisms with a compiler that converts programs to continuation-passing style (CPS). The process of efficiently computing gradients with check pointing requires running, and rerunning, little bits of the program out of order. This is made easier by applying the technique to functional programs. There is a deeper and higher-level message in this talk: machine learning can benefit from advanced techniques from programming language theory.

+

Bio

+

Jeffrey M. Siskind received the B.A. degree in Computer Science from the Technion, Israel Institute of Technology, Haifa, in 1979, the S.M. degree in Computer Science from the Massachusetts Institute of Technology (M.I.T.), Cambridge, in 1989, and the Ph.D. degree in Computer Science from M.I.T. in 1992. He did a postdoctoral fellowship at the University of Pennsylvania Institute for Research in Cognitive Science from 1992 to 1993. He was an assistant professor at the University of Toronto Department of Computer Science from 1993 to 1995, a senior lecturer at the Technion Department of Electrical Engineering in 1996, a visiting assistant professor at the University of Vermont Department of Computer Science and Electrical Engineering from 1996 to 1997, and a research scientist at NEC Research Institute, Inc. from 1997 to 2001. He joined the Purdue University School of Electrical and Computer Engineering in 2002 where he is currently an associate professor. His research interests include computer vision, robotics, artificial intelligence, neuroscience, cognitive science, computational linguistics, child language acquisition, automatic differentiation, and programming languages and compilers.


+ +
Relational Parametricity for Polymorphic Blame Calculus +
+Dustin Jamner +Northeastern University +https://github.com/DIJamner +
+23 June, 2017 12:00PM +WVH 366 +
+

Abstract

+

The polymorphic blame calculus integrates static typing, including universal types, with dynamic typing. The primary challenge with this integration is preserving parametricity: even dynamically-typed code should satisfy it once it has been cast to a universal type. Ahmed et al. (2011) employ runtime type generation in the polymorphic blame calculus to preserve parametricity, but a proof that it does so has been elusive. Matthews and Ahmed (2008) gave a proof of parametricity for a closely related system that combines ML and Scheme, but later found a flaw in their proof. In this work we prove that the polymorphic blame calculus satisfies relational parametricity. The proof relies on a step-indexed Kripke logical relation. The step-indexing is required to make the logical relation well-defined in the case for the dynamic type. The possible worlds include the mapping of generated type names to their concrete types and the mapping of type names to relations. We prove the Fundamental Property of this logical relation and that it is sound with respect to contextual equivalence.

+

Bio

+

Dustin Jamner is an undergraduate at Northeastern University in Computer Science, working with Amal Ahmed.


+ +
CoCoSpec: A Mode-aware Contract Language for Reactive Systems +
+Cesare Tinelli +University of Iowa +http://homepage.cs.uiowa.edu/~tinelli/ +
+1 June, 2017 11:00AM +WVH 366 +
+

Abstract

+

Contract-based software development is a leading methodology for the construction of safety- and mission-critical embedded systems. Contracts are an effective way to establish boundaries between components and can be used efficiently to verify global properties by using compositional reasoning techniques. A contract specifies the assumptions a component makes on its context and the guarantees it provides. Requirements in the specification of a component are often case-based, with each case describing what the component should do depending on a specific situation (or mode) the component is in.

This talk introduces CoCoSpec, a mode-aware assume-guarantee-based contract language for embedded systems. CoCoSpec is built as an extension of the Lustre language and lets users specify mode behavior directly, instead of encoding it as conditional guarantees, thus preventing a loss of mode-specific information. Mode-aware model checkers supporting CoCoSpec can increase the effectiveness of the compositional analysis techniques found in assume-guarantee frameworks and improve scalability. Such tools can also produce much better feedback during the verification process, as well as valuable qualitative information on the contract itself. I will presents the CoCoSpec language and illustrate the benefits of mode-aware model-checking on a case study involving a flight-critical avionics system. The evaluation uses Kind 2, a collaborative, parallel, SMT-based model checker developed at the University of Iowa that provides full support for CoCoSpec.

+

Bio

+

Cesare Tinelli received a Ph.D. in Computer Science from the UIUC in 1999 and is currently a professor in Computer Science at the University of Iowa. His research interests include automated reasoning and formal methods. He has done seminal work in automated reasoning, in particular in Satisfiability Modulo Theories (SMT), a field he helped establish through his research and service activities. His work has been funded both by governmental agencies (AFOSR, AFRL, DARPA, NASA, and NSF) and corporations (Intel, General Electric, Rockwell Collins, and United Technologies) and has appeared in more than 70 refereed publications. He is a founder and coordinator of the SMT-LIB initiative, an international effort aimed at standardizing benchmarks and I/O formats for SMT solvers. He has led the development of the award winning Darwin theorem prover and the Kind model checker. He co-leads the development of CVC4,a widely used and award winning SMT solver, and StarExec, a cross community web-based service for the comparative evaluation of logic solvers.


+ +
Multi-language programming system: a linear experiment +
+ + + +
+5 May, 2017 12:00PM +WVH 366 +
+

Abstract

+

Programming language research keeps inventing more powerful dynamic and static features for programming language. Ideally, thorough language design should be able to propose a single, unified, powerful yet simple view of them as orthogonal aspects of a single programming language that would remain usable by everyone. In practice, it always takes much more time and effort to find the simple/orthogonal presentation than to come up with the desirable features, and we become emotionally attached to programming language monoliths (C++, Scala, GHC Haskell, OCaml, Common Lisp...) that suffer from feature creep. An alternative is to design programming systems as multi-language environment instead. This point of view makes it easier to detect flaws in a proposed programming system design: there is no clear objective definition of what "orthogonal features" really mean, but we have clear ideas of how multi-language systems should work. The Racket Manifesto says: "Each language and component must be able to protect its specific invariants".

In this work we give a formal specification, in terms of full abstraction, for what it means for a new language added to the system to not introduce "abstraction leaks" in the other languages, and discuss scenarios where this translates in (informal) usability properties: having a nice teachable language subset, or gracefully introducing an advanced language for experts. We present a multi-language design that exhibits this formal property by adding a language L with linear types and linear state to a typical ML-style programming language U. L allows to write programs with resource usage guarantees (space consumption, typestate-style safety guarantees) that were not possible in U, without introducing abstraction leaks. We will demonstrate several useful examples, some of them that crucially rely on the fine-grained use of language boundaries, at a subterm level rather than whole-module level.

+

Bio

+


+ +
From Datalog to Flix: A Declarative Language for Fixed Points on Lattices +
+Magnus Madsen +University of Waterloo +plg.uwaterloo.ca/~mmadsen/ +
+14 April, 2017 1:30PM +WVH 366 +
+

Abstract

+

We present FLIX, a declarative programming language for specifying and solving least fixed point problems, particularly static program analyses. FLIX is inspired by Datalog and extends it with lattices and monotone functions. Using FLIX,implementors of static analyses can express a broader range of analyses than is currently possible in pure Datalog, while retaining its familiar rule-based syntax. We define a model-theoretic semantics of FLIX as a natural extension of the Datalog semantics. This semantics captures the declarative meaning of FLIX programs without imposing any specific evaluation strategy. An efficient strategy is semi-naïve evaluation which we adapt for FLIX. We have implemented a compiler and runtime for FLIX, and used it to express several well-known static analyses, including the IFDS and IDE algorithms. The declarative nature of FLIX clearly exposes the similarity between these two algorithms.

This work has previously been presented at PLDI 2016. The talk will cover some of that material as well as recent developments.

+

Bio

+

The speaker is a post doctoral fellow at the University of Waterloo.


+ +
Simple Invariants for proving the safety of distributed protocols and networks +
+Mooly Sagiv +Tel Aviv University +http://www.cs.tau.ac.il/~msagiv/ +
+31 March, 2017 12:00PM +WVH 366 +
+

Abstract

+

Safety of a distributed protocol means that the protocol never reaches a bad state, e.g., a state where two nodes become leaders in a leader-election protocol. Proving safety is obviously undecidable since such protocols are run by an unbounded number of nodes, and their safety needs to be established for any number of nodes. I will describe a deductive approach for proving safety, based on the concept of universally quantified inductive invariants --- an adaptation of the mathematical concept of induction to the domain of programs. In the deductive approach, the programmer specifies a candidate inductive invariant and the system automatically checks if it is inductive. By restricting the invariants to be universally quantified, this approach can be effectively implemented with a SAT solver.

This is a joint work with Ken McMillan (MSR), Oded Padon (TAU), Aurojit Panda(Berkeley) , and Sharon Shoham(TAU) and was integrated into the IVY system: http://www.cs.tau.ac.il/~odedp/ivy/. The work is inspired by Shachar Itzhaky's thesis available from http://people.csail.mit.edu/shachari/

+

Bio

+

Mooly Sagiv is a professor in the School of Computer Sciences at Tel-Aviv University. He is a leading researcher in the area of large scale (inter-procedural) program analysis, and one of the key contributors to shape analysis. His fields of interests include programming languages, compilers, abstract interpretation, profiling, pointer analysis, shape analysis, inter-procedural dataflow analysis, program slicing, and language-based programming environments. Sagiv is a recipient of a 2013 senior ERC research grant for Verifying and Synthesizing Software Composition. Prof. Sagiv served as Member of the Advisory Board of Panaya Inc acquired by Infosys. He received best-paper awards at PLDI'11 and PLDI'12 for his work on composing concurrent data structures and a ACM SIGSOFT Retrospective Impact Paper Award (2011) for program slicing. He is an ACM fellow and a recipient of Microsoft Research Outstanding Collaborator Award 2016.


+ +
Learning to Program and Debug, Automatically +
+Swarat Chaudhuri +Rice University +https://www.cs.rice.edu/~sc40/ +
+24 March, 2017 12:00PM +WVH 366 +
+

Abstract

+

Automating programming and debugging are long-standing goals in computer science. In spite of significant progress in formal methods over the years, we remain very far from achieving these goals. For example, a freshman CS major will typically program circles around today's best program synthesizers. Debugging and verification tools rely on formal specifications, which are hard to provide in many important applications.

Two critical components of the gap between human and machine programmers are that humans learn from experience, i.e., data, and can easily generalize from incomplete problem definitions. In this talk, I will present a general framework for formal methods, based on Bayesian statistical learning, that aims to eliminate these differences. In our framework, descriptions of programming tasks are seen to be "clues" towards a hidden (probabilistic) specification that fully defines the task. Large corpora of real-world programs are used to construct a statistical model that correlates specifications with the form and function of their implementations. The framework can be implemented in a variety of ways, but in particular, through a neural architecture called Bayesian variational encoder-decoders. Inferences made using the framework can be used to guide traditional algorithms for program synthesis and bug-finding.

I will show that this data-driven approach can lead to giant leaps in the scope and performance of automated program synthesis and debugging algorithms. Specifically, I will give a demo of Bayou, a system for Bayesian inductive synthesis of Java programs that goes significantly beyond the state of the art in program synthesis. I will also describe Salento, a debugging system based on our framework that can find subtle violations of API contracts without any kind of specification.

+

Bio

+

Swarat Chaudhuri is an Associate Professor of Computer Science at Rice University. His research lies at the interface of programming systems and artificial intelligence. Much of his recent work is on program synthesis, the problem of automatically generating computer programs from high-level specifications.


+ +
Templates and Types in C++ +
+Jimmy Hartzell +Tower Research + +
+10 March, 2017 12:00PM +WVH 366 +
+

Abstract

+

Like many modern programming languages, C++ supports both generic + programming (templates) and object-oriented programming (inheritance and + virtual functions). Unlike many modern programming languages, the generic + programming mechanism does not fully leverage or interact with the type + system. It amounts to compile-time duck typing: C++ function templates, + like functions from a non-statically typed programming language, have + their full interface implicit in their implementations -- making it hard + to discern what that interface actually is. The concept of Concepts + (compile-time typeclasses) was slated for introduction into C++ in + the C++11 version of the standard, then C++14, then C++17, now C++20... What can +we do to work around their absence?

Relatedly, how do we bridge the gap +between static and dynamic polymorphism? C++ provides both, and sometimes we +want to mix and mash them. Template tricks can also be done to accomplish this +as well.

+

Bio

+

Jimmy Hartzell is a software developer and C++ instructor at Tower Research.


+ +
+

Abstract

+

Topology, when viewed from an unusual perspective (formal topology), describes how to compute with certain objects that are beyond the reach of induction, such as real numbers, probability distributions, streams, and function spaces. Accordingly, one gets a programming language whose types are spaces and whose functions are continuous maps.

Does this language have pattern matching? In functional programming, pattern matching allows definition of a function by partitioning the input and defining the result in each case. We generalize to programming with spaces, where patterns need not represent decidable predicates and also may overlap, potentially allowing nondeterministic behavior in overlapping regions. These overlapping pattern matches are useful for writing a wide array of computer programs on spaces, such as programs that make approximate computations or decisions based on continuous values or that manipulate "partial" datatypes.

This talk will introduce topology from a computational perspective, and explore some programs that can be built with this framework using overlapping pattern matching.

+

Bio

+

Ben Sherman is a second-year PhD student at MIT, advised by Adam Chlipala and Michael Carbin.


+ +
Comparative Study of Generic Programming Features in Object-Oriented Languages +
+Julia Belyakova +Northeastern University +http://staff.mmcs.sfedu.ru/~juliet/index.en.html +
+3 February, 2017 12:00PM +WVH 366 +
+

Abstract

+

Earlier comparative studies of language support for generic programming (GP) have shown that mainstream object-oriented (OO) languages such as C# and Java provide weaker support for GP as compared with functional languages such as Haskell or SML.

One reason is that generic mechanisms of C# and Java are based on F-bounded polymorphism, which has certain deficiencies. Some of these deficiencies are eliminated in more recent languages, such as Scala and Kotlin. However, there are peculiarities of object-oriented languages as a whole, which prevent them from being as expressible as Haskell in terms of generic programming.

In this talk we will cover the main problems of language support for generic programming in C#/Java as well as other modern object-oriented languages, including Scala and Swift. It turns out that all of these languages follow the same approach to constraining type parameters, which leads to inevitable shortcomings. To overcome them, an alternative approach is suggested in several extensions for OO languages, in particular, JavaGenus. The advantages and drawbacks of both approaches will be discussed in the talk.

Here are Julia's Slides.

+

Bio

+

Julia Belyakova graduated in 2014 from Southern Federal University (Rostov-on-Don, Russia) as Master of Science in “Computer Science and Information Technologies”. She is enrolled for PhD program in Southern Federal University and came to Northeastern University as a visiting scientist. She is currently doing research in the area of generic programming for object-oriented languages.


+ +
Making the Browser Reasonable for Sane Programmers +
+John Vilk +University of Massachusetts Amherst +https://jvilk.com/ +
+27 January, 2017 12:00PM +WVH 366 +
+

Abstract

+

Billions of people use browsers to access the web from a variety of devices, but it remains difficult to write and debug the client-side of web applications. The browser exposes unfamiliar, redundant, and incompatible abstractions for a variety of common tasks, preventing developers from adapting existing code written for other platforms. The browser environment is completely event-driven and pervasively concurrent, resulting in complex control flow and race conditions that are difficult to debug.

In this talk, I will describe how my research makes it easier to write and debug web applications. Doppio is a JavaScript runtime system that lets developers run unaltered code written in general-purpose languages, such as Java and C/C++, directly inside the browser. Browsix further enhances Doppio with a kernel, processes, and shared operating system resources, letting multiple off-the-shelf programs concurrently interoperate on the same webpage. ReJS is a low-overhead and high-fidelity time-traveling debugger that lets developers instantaneously step forward and backward through a program's execution. ReJS accurately recreates the application's complete behavior, including the GUI and multiple types of race conditions, while producing miniscule and portable application traces. These resources transform the browser into a first-class application platform.

+

Bio

+

John Vilk is a Ph.D. candidate in the College of Information and Computer Sciences at the University of Massachusetts Amherst. He is a member of the PLASMA lab, and is advised by Emery Berger. John's research aims to improve the browser as an application platform by making it easier to write, debug, and optimize complex web applications. John is a Facebook Fellow (2015), and his work on Doppio: Breaking the Browser Language Barrier is a SIGPLAN Research Highlight (2014) and a PLDI Distinguished Artifact (2014). His research forms the basis of Microsoft ChakraCore's time-traveling debugger and lets millions of people interact with legacy software and games in their browser at the Internet Archive (https://archive.org/). John received his MS from the University of Massachusetts Amherst (2013) and his BS from Worcester Polytechnic Institute (2011).

You can learn more about John at his website, https://jvilk.com/.


+ +
Retractions and Blame +
+Max New +Northeastern University +http://maxsnew.github.io/ +
+15 December, 2016 12:00PM +Behrakis 105 +| Notes
+

Abstract

+

It has long been noted that research on contracts and gradual typing bears a striking resemblance to Dana Scott's work on retracts in domain theory. However, a concrete relationship has been elusive, especially since the notion of *blame* has seemingly no counterpart in domain theory.

We connect these fields by means of gradual type precision, which has been characterized previously using blame in [1]. We propose a definition in terms of section-retraction pairs, allowing us to relate blame and retractions in a precise way. Our definition agrees with [1] on most rules, but differs in a way that suggests a modification to their definition of blame.

[1] Wadler and Findler, Well-typed Programs can't Be Blamed: +http://homepages.inf.ed.ac.uk/wadler/topics/blame.html#blame-esop

+

Bio

+

Max New is a PhD student at Northeastern University working on the semantic foundations of programming languages. He hates proving the same theorem twice and you should too.


+ +
Deciding program equivalence with sums and the empty type +
+Gabriel Scherer +Northeastern University +http://www.ccs.neu.edu/home/gasche/ +
+7 December, 2016 12:00PM +WVH 110 +
+

Abstract

+

The simply-typed λ-calculus, with only function types, is strongly normalizing, and its program equivalence relation is decidable: unlike in more advanced type system with polymorphism or effects, the natural "syntactic" equivalence (βη-equivalence) corresponds to natural "semantic" equivalence (observational or contextual equivalence), and is decidable. Adding product types (pairs) is easy and preserves these properties, but sums (disjoint unions) are, surprisingly, harder. It is only in 1995 that Neil Ghani proved that the equivalence in presence of sums is decidable, and the situation in presence of the empty type (zero-ary sum) was unknown.

We propose an equivalence algorithm for sums and the empty type that takes inspiration from a proof-theoretic technique, named "focusing",designed for proof search. The exposition will be introductory; I will present the difficulties caused by sums and the empty type, present some of the existing approaches for sums in the literature, introduce focusing, and give a high-level intuition of our saturation-based algorithm and its termination argument. No previous knowledge of focusing is assumed. I expect some familiarity with typed lambda-calculi, but will recall all concepts used (for example, βη-equivalence) during the talk.

+

Bio

+

Gabriel is interested in theoretical aspects of type systems, +programming language implementation, general programming language +concepts, and even some syntactic aspects. He has a preference for the +formalizable aspects, or formalizable approaches to programming +language aspects, rather than the often subjective appeal to taste or +intuition.


+ +
Analysis of Android hybrid applications and other fun with WALA +
+Julian Dolby +IBM Research +http://researcher.ibm.com/person/us-dolby +
+1 December, 2016 12:00PM +Richards Hall 140 +
+

Abstract

+

Hybrid apps help developers build multiple apps for different platforms with less duplicated effort, by providing platform-specific functionality via native code and user interactions via javascript code. however, most hybrid apps are developed in multiple programming languages with different semantics, complicating programming. moreover, untrusted javascript code may access device-specific features via native code, exposing hybrid apps to attacks. Unfortunately, there are no existing tools to detect such vulnerabilities. In this paper, we present HybriDroid, the first static analysis framework for Android hybrid apps. First, we investigate the semantics of interoperation of Android Java and JavaScript. Then, we design and implement a static analysis framework that analyzes inter-communication between Android Java and JavaScript. We demonstrate HybriDroid with a bug detector that identifies programmer errors due to the hybrid semantics, and a taint analyzer that finds information leaks cross language boundaries. Our empirical evaluation shows that the tools are practically usable in that they found previously uncovered bugs in real-world Android hybrid apps and possible information leaks via a widely-used advertising platform.

The bulk of this presentation will focus on ASE 2016 work on analysis of hybrid apps (1), a blend of per-platform native code and portable JavaScript. I will also briefly discuss two other recent projects involving WALA: ASE 2015 work on a practically tunable static analysis framework for large-scale JavaScript applications (2), and ISSTA 2015 work on scalable and precise taint analysis for Android (3).

1: Sungho Lee, Julian Dolby, Sukyoung Ryu: HybriDroid: static analysis framework for Android hybrid applications. ASE 2016: 250-261

2: Yoonseok Ko, Hongki Lee, Julian Dolby, Sukyoung Ryu: Practically Tunable Static Analysis Framework for Large-Scale JavaScript Applications (T). ASE 2015: 541-551

3: Wei Huang, Yao Dong, Ana Milanova, Julian Dolby: Scalable and precise taint analysis for Android. ISSTA 2015: 106-117

+

Bio

+

Julian Dolby is a Research Staff Member at the IBM Thomas J. Watson Research Center, where he works on a range of topics, including static program analysis, software testing, concurrent programming models and the semantic Web. He is one of the primary authors of the publically available Watson Libraries for Analysis (WALA) program analysis infrastructure, and his recent WALA work has focused on creating the WALA Mobile infrastructure.

His program analysis work has recently focused on scripting languages like JavaScript and on security analysis of Web applications; this work has been included in IBM products, most notably Rational AppScan, Standard Edition and Source Edition. He was educated at the University of Illinois at Urbana-Champaign as a graduate student where he worked with Professor Andrew Chien on programming systems for massively-parallel machines.


+ +
Sequent calculus as a programming language +
+Zena Ariola +University of Oregon +http://ix.cs.uoregon.edu/~ariola/ +
+28 November, 2016 12:00PM +WVH 110 +
+

Abstract

+

We will present and demonstrate the usefulness of the sequent +calculus as a formal model of computation based on interactions +between producers and consumers of results. This model leads +to a better understanding of call-by-name evaluation by +reconciling the conflicting principles of extensionality and +weak-head evaluation, thus internalizing a known parametricity +result. It allows one to explore two dualities of computation: +the duality between call-by-name and call-by-value, and the +duality between construction and deconstruction. This ultimately +leads to a better linguistic foundation for co-induction as dual +to induction. From a more practical point of view, the sequent +calculus provides a useful inspiration for the design of +intermediate languages.

+

Bio

+

TBD


+ +
Contextual Equivalence of Probabilistic Programs +
+Ryan Culpepper +Northeastern University +http://www.ccs.neu.edu/home/ryanc/ +
+3 November, 2016 12:00PM +WVH 366 +
+

Abstract

+

In this talk I introduce a probabilistic programming language with continuous random variables and soft constraints, define contextual equivalence for the language, and then present a logical relation that is sound with respect to contextual equivalence.

The question of equivalence for probabilistic programs is interesting and difficult because the meaning of a program is a measure defined by integrating over the space of all of the program's possible evaluations. Thus creating a workable logical relation requires proving integral equalities involving intermediate measures that are ``equal enough'' but still respect contextual equivalence.

+

Bio

+

Ryan Culpepper is a research scientist at Northeastern University. He works on probabilistic programming as well as extensible languages and tools for building domain-specific languages. He received his PhD at Northeastern and did a postdoc at the University of Utah.


+ +
Declarative, Convergent Edge Computation +
+Christopher Meiklejohn +Université Catholique +de Louvain +https://christophermeiklejohn.com/ +
+27 October, 2016 12:00PM +WVH 366 +
+

Abstract

+

Consistency is hard and coordination is expensive. As we move into the +world of connected 'Internet of Things' style applications, or +large-scale mobile applications, devices have less power, periods of +limited connectivity, and operate over unreliable asynchronous +networks. This poses a problem with shared state: how do we handle +concurrent operations over shared state, while clients are offline, +and ensure that values converge to a desirable result without making +the system unavailable?

+

We look at a new programming model, called Lasp. This programming +model combines distributed convergent data structures with a dataflow +execution model designed for distribution over large-scale +applications. This model supports arbitrary placement of processing +node: this enables the user to author applications that can be +distributed across data centers and pushed to the edge.

+

Bio

+

Christopher Meiklejohn loves distributed systems and programming +languages. Previously, Christopher worked at Basho Technologies, Inc. +on the distributed key-value store, Riak. Christopher develops a +programming model for distributed computation, called Lasp. +Christopher is currently a Ph.D. student at the Université Catholique +de Louvain in Belgium.


+ +
The Hack Experiment & HHVM, Then And Now +
+Julien Verlaguet & Brett Simmers +Facebook +http://hacklang.org/ +
+25 October, 2016 1:30PM +WVH 366 +
+

Abstract

+

The Hack programming language developed at Facebook is an evolution of PHP. With tens of millions of lines of PHP code in house, the lack of type and poor IDE support were felt to be major threats to Facebook. We designed Hack gradually, adding feature at regular intervals and encouraging developers to adopt them in their code. This talk will touch on the design of the Hack language, its type system, the implementation of the type checker, and the social processes involved in converting 20 millions lines of untyped code to a rich type system.

HHVM is an open source virtual machine for PHP and Hack programs. It was developed by Facebook but is also used by Wikipedia, Baidu, Etsy, and many others. This talk will discuss the history of the project, what goes on behind the scenes while executing your PHP code, and what we're currently working on.

+

Bio

+

Julien Verlaguet is a OCaml programmer with a Master from Université Pierre et Marie Curie in Paris. Before joining Facebook in 2011, he worked at Esterel technologies on verified compilation. At Facebook he designed the Hack programming language, a typed offspring of PHP and managed to convince the majority of developers in the company to add types to their code.

Brett Simmers is a Software Engineer at Facebook, where he's been working on HHVM for the past five years. He primarily works on the internals of the JIT compiler and HHVM's profile-guided optimizations. Before joining Facebook he spent two years at VMware working on the Linux version of VMware Workstation.


+ +
Implementing a Functional Language for Flix +
+Ming-Ho Yee +Northeastern University +http://mhyee.com +
+20 October, 2016 12:00PM +WVH 366 +
+

Abstract

+

Static program analysis is a powerful technique for maintaining software, with applications such as compiler optimizations, code refactoring, and bug finding. Static analyzers are typically implemented in general-purpose programming languages, such as C++ and Java; however, these analyzers are complex and often difficult to understand and maintain. An alternate approach is to use Datalog, a declarative language. Implementors can express analysis constraints declaratively, which makes it easier to understand and ensure correctness of the analysis. Furthermore, the declarative nature of the analysis allows multiple, independent analyses to be easily combined.

+

Flix is a programming language for static analysis, consisting of a logic language and a functional language. The logic language is inspired by Datalog, but supports user-defined lattices. The functional language allows implementors to write functions, something which is not supported in Datalog. These two extensions, user-defined lattices and functions, allow Flix to support analyses that cannot be expressed by Datalog, such as a constant propagation analysis. Datalog is limited to constraints on relations, and although it can simulate finite lattices, it cannot express lattices over an infinite domain. Finally, another advantage of Flix is that it supports interoperability with existing tools written in general-purpose programming languages.

+

This talk provides an overview of the implementation and evaluation of the Flix functional language. The implementation involves abstract syntax tree transformations, an interpreter back-end, and a code generator back-end, and must support a number of interesting language features, such as pattern matching, first-class functions, and interoperability. The evaluation compares the interpreter and code generator back-ends in terms of correctness and performance.

+

Bio

+

Ming-Ho Yee is a Ph.D. student at Northeastern University. He works on programming language design and implementation with Jan Vitek. He received a Bachelor of Software Engineering from the University of Waterloo, and then continued for a Master of Mathematics in Computer Science under the supervision of Ondřej Lhoták.


+ +
Behavioral Non-Portability in Decision-Making Programs +
+Thomas Wahl +Northeastern University +http://www.ccs.neu.edu/home/wahl/ +
+13 October, 2016 12:00PM +WVH 366 +
+

Abstract

+

The precise semantics of floating-point arithmetic programs depends on the execution platform, including the compiler and the target hardware. Such platform dependencies infringe on the highly desirable goal of software portability (which is in fact promised by heterogeneous computing frameworks like OpenCL): the same program run on the same inputs on different platforms can produce different results. In other words, portability does not guarantee reproducibility, and this is a more or less accepted state of affairs.

+

Serious doubts on the portability of numeric applications arise when differences in results are behavioral, i.e. when they lead to changes in the control flow of a program. In this talk I will first present an algorithm that takes a numeric procedure and determines whether a given input can lead to different decisions depending merely on how the arithmetic in the procedure is compiled and executed. I will then show how this algorithm can be used in static and dynamic analyses of programs, to estimate their numeric stability. I will illustrate the results on examples characteristic of numeric computing where control flow divergence actually occurs across different execution platforms.

+

Joint with Yijia Gu, Mahsa Bayati, and Miriam Leeser, Northeastern University, Boston, USA

+

Bio

+

Thomas Wahl joined the faculty of Northeastern University in 2011. His research concerns the reliability (whatever that means) of complex computing systems. Two domains notorious for their fragility are concurrency and numerical computing. With colleagues, Wahl has investigated how floating-point arithmetic can "hijack" a program's computation when run on non-standard architectures, such as heterogeneous and custom-made embedded platforms. You will witness some hijacking attempts in the talk today.


+ +
Concolic Testing: A Decade Later +
+Koushik Sen +University of California, Berkeley +https://www.cs.berkeley.edu/~ksen/ +
+6 October, 2016 12:00PM +WVH 366 +
+

Abstract

+

Symbolic execution, which was introduced more than four decades ago, is typically used in software testing to explore as many different program paths as possible in a given amount of time, and for each path to generate a set of concrete input values exercising it, and check for the presence of various kinds of errors including assertion violations, uncaught exceptions, security vulnerabilities, and memory corruption. A key limitation of classical symbolic execution is that it cannot generate useful test inputs if the program under test uses complex operations such as pointer manipulations and non-linear arithmetic operations.

+

Our research on Concolic Testing (also known as DART: Directed Automated Random Testing or Dynamic Symbolic Execution) alleviated the limitations of classical symbolic execution by combining concrete execution and symbolic execution. We demonstrated that concolic testing is an effective technique for generating high-coverage test suites and for finding deep errors in complex software applications. The success of concolic testing in scalably and exhaustively testing real-world software was a major milestone in the ad hoc world of software testing and has inspired the development of several industrial and academic automated testing and security tools.

+

One of the key challenges of concolic testing is the huge number of programs paths in all but the smallest programs, which is usually exponential in the number of static branches in the code. In this talk I will describe MultiSE, a new technique for merging states incrementally during symbolic execution, without using auxiliary variables. The key idea of MultiSE is based on an alternative representation of the state, where we map each variable, including the program counter, to a set of guarded symbolic expressions called a value summary. MultiSE has several advantages over conventional symbolic execution and state merging techniques: 1) value summaries enable sharing of symbolic expressions and path constraints along multiple paths, 2) value-summaries avoid redundant execution, 3) MultiSE does not introduce auxiliary symbolic values, which enables it to make progress even when merging values not supported by the constraint solver, such as floating point or function values. We have implemented MultiSE for JavaScript programs in a publicly available open-source tool. Our evaluation of MultiSE on several programs shows that MultiSE can run significantly faster than traditional symbolic execution.

+

Bio

+

Koushik Sen is an associate professor in the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley. His research interest lies in Software Engineering, Programming Languages, and Formal methods. He is interested in developing software tools and methodologies that improve programmer productivity and software quality. He is best known for his work on ³DART: Directed Automated Random Testing² and concolic testing. He has received a NSF CAREER Award in 2008, a Haifa Verification Conference (HVC) Award in 2009, a IFIP TC2 Manfred Paul Award for Excellence in Software: Theory and Practice in 2010, a Sloan Foundation Fellowship in 2011, a Professor R. Narasimhan Lecture Award in 2014, and an Okawa Foundation Research Grant in 2015. He has won several ACM SIGSOFT Distinguished Paper Awards. He received the C.L. and Jane W-S. Liu Award in 2004, the C. W. Gear Outstanding Graduate Award in 2005, and the David J. Kuck Outstanding Ph.D. Thesis Award in 2007, and a Distinguished Alumni Educator Award in 2014 from the UIUC Department of Computer Science. He holds a B.Tech from Indian Institute of Technology, Kanpur, and M.S. and Ph.D. in CS from University of Illinois at Urbana-Champaign.


+ +
LaCasa: Lightweight Affinity and Object Capabilities in Scala +
+Philipp Haller +KTH Royal Institute of Technology +http://www.csc.kth.se/~phaller/ +
+29 September, 2016 12:30PM +WVH 366 +
+

Abstract

+

Aliasing is a known source of challenges in the context of imperative object-oriented languages, which have led to important advances in type systems for aliasing control. However, their large-scale adoption has turned out to be a surprisingly difficult challenge. While new language designs show promise, they do not address the need of aliasing control in existing languages.

+

This paper presents a new approach to isolation and uniqueness in an existing, widely-used language, Scala. The approach is unique in the way it addresses some of the most important obstacles to the adoption of type system extensions for aliasing control. First, adaptation of existing code requires only a minimal set of annotations. Only a single bit of information is required per class. Surprisingly, the paper shows that this information can be provided by the object-capability discipline, widely-used in program security. We formalize our approach as a type system and prove key soundness theorems. The type system is implemented for the full Scala language, providing, for the first time, a sound integration with Scala’s local type inference. Finally, we empirically evaluate the conformity of existing Scala open-source code on a corpus of over 75,000 LOC.

+

Bio

+

Philipp Haller is an assistant professor in the theoretical computer science group at KTH Royal Institute of Technology, the leading technical university in Sweden. His main research interests are programming languages, type systems, concurrent, and distributed programming. Philipp is co-author of Scala's async/await extension for asynchronous computations, and one of the lead designers of Scala's futures and promises library. As main author of the book "Actors in Scala," he created Scala's first widely-used actors library. Philipp was co-chair of the 2013 and 2014 editions of the Scala Workshop, and co-chair of the 2015 ACM SIGPLAN Scala Symposium. Previously, he has held positions at Typesafe, Stanford University, and EPFL. He received a PhD in computer science from EPFL in 2010.


+ +
Performance in Julia +
+Jan Vitek +Northeastern University +http://janvitek.org/ +
+15 September, 2016 12:30PM +WVH 366 +
+

Abstract

+

Julia, like R, is a dynamic language for scientific computing but, unlike R, it was explicitly designed to deliver performance competitive to traditional batch-compiled languages. To achieve this Julia's designers made a number of unusual choices, including the presence of a set of type annotations that are used for dispatching methods and speed up code, but not for type-checking. The result is that many Julia programs are competitive with equivalent programs written in C. This talk gives a brief overview of the key points of Julia's design and considers whether similar ideas could be adopted in R.

+

Bio

+

Jan Vitek, "http://janvitek.org/", is a Professor at Northeastern University CCIS. He works on the design and implementation of programming abstractions for areas and in languages that are *terrifying*. For instance, he has worked on a real-time JVM for running Java on an actual embedding real-time system, on understanding JavaScript to make it more secure, and on getting R (yes R) to be scalable for data analysis. Apparently he does these things because they are used in the "real world" to "solve problems". He also has an excellent sense of humor and didn't give me a bio, so I took some liberties and hope he doesn't mind.


+ +
A Categorical Perspective on Type Refinement +
+Noam Zeilberger +Inria Ecole Polytechnique +http://noamz.org/ +
+13 July, 2016 11:45AM +WVH 366 +
+

Abstract

+

A "type refinement system" is a type system built on top of +a typed programming language, as an extra layer of typing. Over the +past few years Paul-André Melliès and I have investigated a +categorical foundation for type refinement systems, which is based on +the idea of looking at a type refinement system "backwards", in terms +of the functor which erases the extra layer of types. The talk will +give an introduction to this framework as well as to some of the +insights that it offers, such as:

+

* The idea of viewing Hoare logic as a type refinement system

+

* The importance of the bifibrational operations of "pushing forward" and + "pulling back" (which generalize strongest postconditions and weakest preconditions)

+

* How those operations can be combined with a monoidal closed structure, to + model type refinement systems built over pure lambda calculus

+

I will not assume any prior background in category theory, although as background reading audience +members may be interested in the lecture notes for my recent course at OPLSS +(http://noamz.org/oplss16/refinements-notes.pdf, especially Ch.3).

+

Bio

+Since receiving his PhD from Carnegie Mellon in 2009, Noam Zeilberger has been actively pursuing a career as an itinerant postdoc, with gigs in Paris (Laboratoire PPS), Madrid (IMDEA Software), Princeton (IAS), and back in Paris (MSR-Inria). He is currently a member of Inria Team Parsifal at Ecole Polytechnique. In his free time, he enjoys enumerating lambda terms, and his favorite combinators are B, C and I.

+ +
Leveraging Fine-grained Data Flows in Web Applications +
+James Mickens +Harvard University +http://mickens.seas.harvard.edu/ +
+5 July, 2016 1:30PM +WVH 366 +
+

Abstract

+A modern web page contains megabytes of HTML, CSS, images, and JavaScript. Loading such a page requires a browser to evaluate a complex dependency graph involving those resources; once the page is loaded, subsequent interactions between those resources and the user can lead to tricky-to-diagnose bugs. In this talk, I'll describe how tracking fine-grained data flows can allow us to reduce page load times by prioritizing the loads of the highest ancestors in the data flow graph. I'll also describe initial work in using data flows to assist with time-travel debugging (in which developers use a logging-and-replay framework to analyze buggy program executions). +

Bio

+James Mickens is an IEEE Knight of the Republic, an ACM Templar for Non-Open Access, and a Royal Proceeding of Her Majesty’s Royal Proceedings. His appreciation for syntactically correct code has led him to be called “a semicolon in human form.” His online shopping habits have too many dimensions to be k-means clustered, so he is only shown ads about dinosaurs and ancient siege machines. This does not bother James Mickens, and explains why he spends his summers attacking France with triceratops horns.

+ +
New language ideas for user-defined side-effects: algebraic effect handlers. +
+Gabriel Scherer +Northeastern University +http://gallium.inria.fr/~scherer/ +
+18 May, 2016 11:45AM +WVH 366 +
+

Abstract

+This talk reports on some cool ideas I learned during a week-long +seminar in Dagstuhl last month; it will present other people's work, +not my own. It is aimed at an audience interested in programming +language design in general, but not familiar with the theoretical +treatment of pure languages and side-effects. + +We shall start with *demos* first, explaining short code examples in +three programming languages: ML with exceptions as a warmup, then the +new language 'Eff' of Matija Pretnar and Andrej Bauer, that first +implemented so-called effect handlers, and finally the language +'Frank' of Conor McBride, that integrates these handlers in a more +uniform style of "effectful call-by-value programming". + +Only second will we discuss some *theory*, in an accessible way: +monads and algebraic effects, which are two distinct ways to formalize +side-effects, and the difference between "direct" and "indirect" style +of effectful programming. + +Underlying this talk are two larger, important questions of +programming language design, that will be touched during the talk and +we can discuss further afterwards: +- Do programming language need a facility for user-defined side-effects? +- When should we encode new design ideas as libraries/macros in an + expressive language, and when should we design languages afresh for + them? +

Bio

+Gabriel is interested in theoretical aspects of type systems, +programming language implementation, general programming language +concepts, and even some syntactic aspects. He has a preference for the +formalizable aspects, or formalizable approaches to programming +language aspects, rather than the often subjective appeal to taste or +intuition.

+ +
A Coq Library For Internal Verification of Running-Times +
+Jay McCarthy +U. Massachusetts, Lowell +https://jeapostrophe.github.io/home/ +
+17 May, 2016 1:30PM +WVH 366 +
+

Abstract

+We present a Coq library that lifts an abstract yet precise +notion of running-time into the type of a function. Our library is +based on a monad that counts abstract steps, controlled by one of the +monadic operations. The monad's computational content, however, is +simply that of the identity monad so programs written in our +monad (that recur on the natural structure of their arguments) extract +into idiomatic OCaml code. We evaluated the expressiveness of the +library by proving that red-black tree insertion and search, merge +sort, insertion sort, Fibonacci, iterated list insertion, BigNum +addition, and Okasaki's Braun Tree algorithms all have their expected +running times. +

Bio

+Jay McCarthy is an associate professor at UMass Lowell in the Computer +Science Department. He is a member of the PLT research group and works +on the Racket programming language. He is passionate about computer +science education & diversity, formal verification, programming +language expressiveness, and his wonderful family.

+ +
Internet of Things: From Small- to Large-Scale Orchestration +
+Charles Consel +Bordeaux Institute of Technology +http://phoenix.inria.fr/charles-consel +
+2 May, 2016 11:00AM +WVH 366 +
+

Abstract

+The domain of Internet of Things (IoT) is rapidly expanding beyond research and becoming a major industrial market with such stakeholders as major manufacturers of chips and connected objects, and fast-growing operators of low-power wide-area networks. Importantly, this emerging domain is driven by applications that leverage the infrastructure to provide users with innovative, high-value services. Because of this application-centric approach, software development plays a key role to realize the full potential of IoT. + +In this talk, we argue that there is a continuum between orchestrating connected objects in the small and in the large, fostering a unified approach to application development. We examine the requirements for orchestrating connected objects and address them with domain-specific design concepts. We then show how to map these design concepts into dedicated programming patterns and runtime mechanisms. + +Our work revolves around domain-specific notations integrated into a tool-based design methodology, dedicated to develop IoT applications. We have applied our work across a spectrum of infrastructure sizes; we present examples, ranging from an automated pilot in avionics, to an assisted living platform for the home of seniors, to a parking management system in a smart city. +

Bio

+Charles Consel is a professor of Computer Science at Bordeaux Institute of Technology. He served on the faculty of Yale University, Oregon Graduate Institute and the University of Rennes. + +His research contributions cover programming languages, software engineering, operating systems, pervasive computing, and assistive computing. + +He leads the Phoenix group at Inria that conducts multi-disciplinary research to design, develop, deploy and assess assistive computing support. This research combines (1) Cognitive Science to study user needs and make a rigorous assessment of assistive services; (2) Sensing and actuating expertise to support the user, based on accurate and rich interactions with the environment; (3) Computer Science to support and guide the development process of the assistive services.

+ +
STOKE: Search-Based Compiler Optimization +
+Alex Aiken +Stanford University +https://theory.stanford.edu/~aiken/ +
+15 April, 2016 11:00AM +WVH 366 +
+

Abstract

+The optimization component of a compiler takes a program as input and +produces another, hopefully faster, program as output. At a high level, +optimizers solve a search problem over a space of programs, but the +traditional architecture of a compiler's optimizer does no search at +all. This talk will present the STOKE project, which is exploring the +use of Monte Carlo search methods as the basis of a modern compiler +optimizer. We will show that search-based program optimization can +consistently improve, sometimes substantially, on current production +compilers at their highest levels of optimization and can even compete +with expert, hand-written assembly. We will also discuss the unique +and challenging verification problems that arise when a compiler +produces code using a random process. +

Bio

+Alex Aiken is the Alcatel-Lucent Professor and the current +Tencent Chair of the Computer Science Department at Stanford. Alex +received his Bachelors degree in Computer Science and Music from +Bowling Green State University in 1983 and his Ph.D. from Cornell +University in 1988. Alex was a Research Staff Member at the IBM +Almaden Research Center (1988-1993) and a Professor in the EECS +department at UC Berkeley (1993-2003) before joining the Stanford +faculty in 2003. His research interest is in areas related to +programming languages. He is an ACM Fellow, a recipient of Phi Beta +Kappa's Teaching Award, and a former National Young Investigator.

+ +
The Secret Life of Mobile Applications +
+Julia Rubin +MIT +https://people.csail.mit.edu/mjulia/ +
+5 April, 2016 10:30AM +WVH 366 +
+

Abstract

+As software becomes increasingly more complex and yet more pervasive, poor understanding of software behavior compromises the quality and the integrity of software systems that we use. In this talk, I will show that automated analysis techniques can help to identify and reason about software behavior characteristics that matter to humans. After a brief overview of my current research directions, I will focus on techniques for identifying privacy violations in mobile applications, i.e., leakages of sensitive information such as user location and shopping preferences. I will present a set of solutions that rely on contextual, functional and usage-based clues for improving the accuracy of leakage detection and for distinguishing between “legitimate” and “illegitimate” information distribution patterns. +

Bio

+Julia Rubin is a Postdoctoral Researcher in the EECS department at MIT. Prior to that, she was a Research Staff Member and, part of the time, a manager at IBM Research in Haifa, Israel. She received her PhD in Computer Science from the University of Toronto, Canada in 2014. Julia’s research interests are in software engineering, program analysis and software security, focusing on improving the quality and the integrity of modern software systems. Her recent work in this area won an ACM Distinguished Paper Award at ASE, two Best Paper Awards, at SPLC and CSMR, and was nominated for Facebook’s Internet Defense Prize at the USENIX Security Symposium.

+ +
Reasoning about Object Capabilities with Logical Relations and Effect Parametricity +
+Dominique Devriese +Katholieke Universiteit Leuven +https://distrinet.cs.kuleuven.be/people/dominiqu +
+25 February, 2016 1:30PM +WVH 366 +
+

Abstract

+Object capabilities are a technique for fine-grained privilege separation in programming languages and systems,with important applications in security. However, current formal characterisations do not fully capture capability-safety of a programming language and are not sufficient for verifying typical applications. Using state-of-the-art techniques from programming languages research, we define a logical relation for a core calculus of JavaScript that better characterises capability-safety. The relation is powerful enough to reason about typical capability patterns and supports evolvable invariants on shared data structures, capabilities with restricted authority over them and isolated components with restricted communication channels. We use a novel notion of effect parametricity for deriving properties about effects. We demonstrate that our results imply memory access bounds that have previously been used to characterise capability-safety. +

Bio

+Dominique is a postdoctoral researcher in the research group DistriNet, part of the Computer Science department of the Katholieke Universiteit Leuven. He holds a postdoctoral research fellowship of the Research Foundation - Flanders (FWO). He works on formalising properties of object-oriented and object-capability programming languages---specifically a property called effect parametricity--is the author of the grammar-combinators Haskell parsing library, and has added instance arguments to the programming language/proof assistant Agda. He is interested in information flow security, secure compilation,full abstraction, and functional and dependently typed programming languages.

+ +
JavaScript in the Small +
+Satish Chandra +Samsung +https://sites.google.com/site/schandraacmorg/ +
+22 February, 2016 11:00AM +WVH 366 +
+

Abstract

+Emerging consumer electronics products are running the same software platforms that power smartphones. This leads to the appealing idea that a uniform programming abstraction can be used for app development for a range of devices, from wearables to smartphones. In practice, however, devices wary in their hardware capabilities and this has an impact on app development. In this talk, I will focus on ways in which JavaScript may be used and/or run differently to accommodate the different hardware capabilities across devices. Specifically, I will present a subset of JavaScript that strikes a balance between retaining the flavor of JavaScript (e.g. use of prototype inheritance, no explicit type declarations) and the possibility of ahead-of-time compilation to efficient code. +

Bio

+Satish Chandra obtained a PhD from the University of Wisconsin-Madison in 1997, and a B.Tech from the Indian Institute of Technology-Kanpur in 1991, both in computer science. From 1997 to 2002, he was a member of technical staff at Bell Laboratories, where his research focused on program analysis, domain-specific languages, and data-communication protocols. From 2002 to 2013, he was a research staff member at IBM Research, where his research focused on bug finding and verification, software synthesis, and test automation. He joined Samsung Electronics in 2013, where he leads the advanced programming tools research team. He is an ACM Distinguished Scientist.

+ +
Journey to Find Bugs in Real-World Web Applications in the Wild +
+Sukyoung Ryu +Korea Advanced Institute of Science and Technology +http://plrg.kaist.ac.kr/ryu +
+19 February, 2016 11:45AM +WVH 366 +
+

Abstract

+Analyzing real-world web applications is a challenging task. On top of understanding the semantics of JavaScript, it requires modeling of web documents, platform objects, and interactions between them. Most web applications load JavaScript code dynamically, which makes pure static analysis approaches inapplicable. In this talk, we present our journey to analyze JavaScript web applications in the wild using various approaches. From pure JavaScript programs to JavaScript web applications using platform-specific libraries, we explain technical challenges in analyzing each of them and how we built an open-source analysis framework for JavaScript, SAFE, that addresses the challenges incrementally. Finally, we discuss our ongoing work on analysis of Android hybrid applications. +

Bio

+

+ +
Taming release-acquire consistency +
+Ori Lahav +Max Planck Institute for Software Systems +https://www.mpi-sws.org/~orilahav/ +
+27 January, 2016 11:45AM +WVH 366 +
+

Abstract

+

Multiprocessors and concurrent programming are now pervasive. Typically, they do not guarantee sequential consistency (a.k.a. interleaving semantics), which is the standard assumption by most work on semantics and verification. Instead, they employ subtle memory models, exposing unexpected relaxed behaviors arising from hardware and compiler optimizations.

+

In this talk, I will focus on one such model --- the release-acquire fragment of the C/C++11 memory model. I will describe its merits, and show how it can be further improved, without additional implementation costs, to: (i) forbid dubious behaviors that are not observed in any implementation; (ii) support fence instructions that restore sequential consistency; and (iii) admit an equivalent intuitive operational semantics.

+

The talk is based on a joint work with Nick Giannarakis and Viktor Vafeiadis, to be presented in POPL'16.

+

Bio

+Ori Lahav is a postdoctoral researcher at MPI-SWS. He obtained his PhD from Tel Aviv University in the area of non-classical logics. His current research interests are in programming languages generally, with specific focus on memory models, concurrency, verification, and logic.

+ +
Secure Compilation to Protected Module Architectures +
+Marco Patrignani +Max Planck Institute for Software Systems +http://www.mpi-sws.org/~marcopat/ +
+25 January, 2016 11:00AM +WVH 366 +
+

Abstract

+This talk will informally describe protected module architectures (PMA), a security architecture that provides an assembly-level memory isolation mechanism. Then it will describe how to devise a secure (fully-abstract) compiler for an object-oriented language to PMA. Finally, it will present how to prove the said compiler to be secure and discuss open, related research trajectories. +

Bio

+Marco Patrignani did his bachelor and masters study in Bologna, then he obtained a Ph.D. in computer science from the KU Leuven, Belgium. There, with his advisors Dave Clarke and Frank Piessens, he studied secure (fully-abstract) compilation for Intel-SGX like architectures, i.e., untyped assembly languages extended with a memory isolation mechanism. Additionally, he investigated trace-based characterisation of the behaviour of those architectures. He is now a post-doc at MPI-SWS, Germany working on more secure-compilation-related topics with Deepak Garg.

+ +
Program verification under weak memory consistency +
+Viktor Vafeiadis +Max Planck Institute for Software Systems +http://www.mpi-sws.org/~viktor/ +
+14 January, 2016 1:30PM +WVH 366 +
+

Abstract

+

Weak memory models formalize the inconsistent behaviors that one can observe in multithreaded programs running on modern hardware. In so doing, they complicate the already-difficult task of reasoning about correctness of concurrent code. Worse, they render impotent most formal methods that have been developed to tame concurrency, which almost universally assume a strong (i.e., sequentially consistent) memory model. In response, we have developed a number of alternative reasoning techniques that are sound for programs running weak memory consistency. I will cover both program logics, such as relaxed separation logic, as well as theorems that allow reducing reasoning about well-structured weakly consistent implementations down to sequential consistency, and show how these can be applied to reason about a practical RCU implementation.

+

Bio

+Viktor Vafeiadis is a tenure-track researcher at MPI-SWS. He received his BA (2004) and PhD (2008) from the University of Cambridge. Before joining MPI-SWS in October 2010, he was a postdoc at Microsoft Research and at the University of Cambridge. He is interested in programming languages and verification with a focus program logics for weak memory, program logics for concurrency, compiler verifications, automated verification of concurrent programs, and interactive theorem proving.

+ +
Lightweight Formal Methods for LLVM Verification +
+Santosh Nagarakatte +Rutgers University +http://www.cs.rutgers.edu/~santosh.nagarakatte/ +
+13 January, 2016 12:00PM +WVH 366 +
+

Abstract

+

Compilers form an integral component of the software +development ecosystem. Compiler writers must understand the +specification of source and target languages to design sophisticated +algorithms that transform programs while preserving +semantics. Unfortunately, compiler bugs in mainstream compilers are +common. Compiler bugs can manifest as crashes during compilation, or, +much worse, result in the silent generation of incorrect +programs. Such mis-compilations can introduce subtle errors that are +difficult to diagnose and generally puzzling to software +developers.

+

The talk will describe the problems in developing peephole +optimizations that perform local rewriting to improve the efficiency +of LLVM code. These optimizations are individually difficult to get +right, particularly in the presence of undefined behavior; taken +together they represent a persistent source of bugs. The talk will +present Alive, a domain-specific language for writing optimizations +and for automatically either proving them correct or else generating +counterexamples. A transformation in Alive is shown to be correct +automatically by encoding the transformation into constraints, which +are automatically checked for validity using a Satisfiability Modulo +Theory (SMT) solver. Furthermore, Alive can be automatically +translated into C++ code that is suitable for inclusion in an LLVM +optimization pass.

+

Alive is based on an attempt to balance usability and formal +methods; for example, it captures—but largely hides—the detailed +semantics of three different kinds of undefined behavior in LLVM. We +have translated more than 300 LLVM optimizations into Alive and, in +the process, found that eight of them were wrong. I will conclude the +talk highlighting the lessons learned and the challenges in +incorporating lightweight formal methods in the tool-chain of the +compiler developer.

+

Bio

+Santosh Nagarakatte is an Assistant Professor of Computer Science at Rutgers University. He obtained his PhD from the University of Pennsylvania. His research interests are in Hardware-Software Interfaces spanning Programming Languages, Compilers, Software Engineering, and Computer Architecture. His papers have been selected as IEEE MICRO TOP Picks papers of computer architecture conferences in 2010 and 2013. He has received the NSF CAREER Award in 2015, PLDI 2015 Distinguished Paper Award, and the Google Faculty Research Award in 2014 for his research on incorporating lightweight formal methods for LLVM compiler verification.

+ +
Feature Specific Profiling for R +
+Leif Andersen +Northeastern University +http://leifandersen.net +
+2 December, 2015 12:00PM +WVH 366 +
+

Abstract

+

Programmers use profilers to understand the performance +characteristics of +their programs and to focus on those pieces whose improvement may yield the +largest gains. A conventional profiler measures the time that a program +spends in functions, methods, expressions, and statements. Racket's novel +feature-specific profiler supplements this information with timings of +instances of linguistic features. This paper reports the results of a +successful reproducibility effort to adapt feature-specific profiling to +the R programming language. Specifically, the paper demonstrates how easy +and effective it is to add the necessary language support, that the +approach usefully enhances the information produced by a classical +profiler, and that the additional overhead is tolerable.

+

Bio

+Leif Andersen is a second year Ph.D. student at +Northeastern University, studying programming language with Matthias +Felleisen.

+ +
Performance Matters +
+Emery Berger +U. Massachusetts, Amherst +http://emeryberger.com/ +
+20 November, 2015 10:30AM +WVH 366 +
+

Abstract

+

Performance clearly matters to users. The most common software update +on the AppStore *by far* is "Bug fixes and performance enhancements." +Now that Moore's Law Free Lunch has ended, programmers have to work +hard to get high performance for their applications. But why is +performance so hard to deliver?

+

I will first explain why our current approaches to evaluating and + optimizing performance don't work, especially on modern hardware and + for modern applications. I will then present two systems that address + these challenges. Stabilizer is a tool that enables statistically + sound performance evaluation, making it possible to understand the + impact of optimizations and conclude things like the fact that the -O2 + and -O3 optimization levels are indistinguishable from noise + (unfortunately true).

+

Since compiler optimizations have largely run out of steam, we need + better profiling support, especially for modern concurrent, + multi-threaded applications. Coz is a novel "causal profiler" that + lets programmers optimize for throughput or latency, and which + pinpoints and accurately predicts the impact of optimizations. Coz's + approach unlocks numerous previously unknown optimization + opportunities. Guided by Coz, we improved the performance of Memcached + by 9%, SQLite by 25%, and accelerated six Parsec applications by as + much as 68%; in most cases, these optimizations involved modifying + under 10 lines of code.

+

This talk is based on work with Charlie Curtsinger published at ASPLOS + 2013 (Stabilizer) and SOSP 2015 (Coz), where it received a Best Paper + Award.

+

Bio

+

Emery Berger is a Professor in the College of Information and Computer +Sciences at the University of Massachusetts Amherst, the flagship +campus of the UMass system. He graduated with a Ph.D. in Computer +Science from the University of Texas at Austin in 2002. Professor +Berger has been a Visiting Scientist at Microsoft Research (7 times) +and at the Universitat Politecnica de Catalunya (UPC) / Barcelona +Supercomputing Center (BSC).

+

Professor Berger’s research spans programming languages, runtime +systems, and operating systems, with a particular focus on systems +that transparently improve reliability, security, and performance. He +is the creator of a number of influential software systems including +Hoard, a fast and scalable memory manager that accelerates +multithreaded applications (used by companies including British + Telecom, Cisco, Crédit Suisse, Reuters, Royal Bank of Canada, SAP, and + Tata, and on which the Mac OS X memory manager is based); DieHard, an +error-avoiding memory manager that directly influenced the design of +the Windows 7 Fault-Tolerant Heap; and DieHarder, a secure memory +manager that was an inspiration for hardening changes made to the +Windows 8 heap.

+

His honors include a Microsoft Research Fellowship, an NSF CAREER +Award, a Lilly Teaching Fellowship, a Most Influential Paper Award at +OOPSLA 2012, a Google Research Award, a Microsoft SEIF Award, a Best +Artifact Award at PLDI, and Best Paper Awards at FAST, OOPSLA, and +SOSP; he was named an ACM Senior Member in 2010. Professor Berger is +currently a Member of the SIGPLAN Executive Committee and an Associate +Editor of the ACM Transactions on Programming Languages and Systems, +and will serve as Program Chair for PLDI 2016.


+ +
Programming Languages Meets Programmable Networks +
+Arjun Guha +U. Massachusetts, Amherst +https://people.cs.umass.edu/~arjun/home/ +
+16 November, 2015 1:00PM +WVH 366 +
+

Abstract

+

Computer networks do not simply connect machines together, but run several + applications on network devices, such as load balancers, intrusion detection + systems, authentication portals, and more. Historically, these applications were + black-boxes running on proprietary hardware, but software-defined networking + (SDN) now allows anyone to write their own programs using open networking + protocols (e.g., OpenFlow). So, what are the right abstractions for programming networks? This talk will try + to address this question in three ways.

+

First, we present a syntactic theory of network forwarding called NetKAT, which supports equational reasoning about network-wide behavior. Using NetKAT, programmers can ask and answer questions like, "Can A communicate with B?", + "Does all traffic traverse my intrusion detection system?", "Is there a loop in +my network?", and so on.

+

Second, we present a fast and efficient compiler for NetKAT. Although several +network compilers already exist, they are unusable on even moderately sized +networks. Using new data structures and compilation algorithms, our new compiler +is two orders of magnitudes faster than prior work and scales to large +datacenter networks.

+

Finally, we consider the problem of building a reliable runtime system for +NetKAT. NetKAT abstracts away several low-level details of networking hardware. +Although this is a boon for the network programmer, the burden now shifts to us +to engineer abstractions correctly. We present a Coq-certified runtime system +that is proven correct with respect to a detailed operational model software- +defined networks.

+

Bio

+Arjun Guha is an assistant professor of Computer Science at UMass Amherst. He + enjoys tackling problems in systems using the tools and principles of + programming languages. Apart from network programming, he has worked on Web + security and system configuration languages. He received a PhD in Computer + Science from Brown University in 2012 and a BA in Computer Science from Grinnell + College in 2006.

+ +
Declarative Programming for Eventual Consistency +
+Suresh Jagannathan +Purdue University +https://www.cs.purdue.edu/homes/suresh/ +
+13 November, 2015 10:30AM +WVH 366 +
+

Abstract

+

In geo-replicated distributed data stores, the need to ensure responsiveness + in the face of network partitions and processor failures results in + implementations that provide only weak (so-called eventually consistent) + guarantees on when data updated by one process becomes visible to another. + Applications must be carefully constructed to be aware of unwanted + inconsistencies permitted by such implementations (e.g., having negative + balances in a bank account, or having an item appear in a shopping cart + after it has been removed), but must balance correctness concerns with + performance and scalability needs. Because understanding these tradeoffs + requires subtle reasoning and detailed knowledge about the underlying data + store, implementing robust distributed applications in such environments is + often an error-prone and expensive task.

+ +

To overcome these issues, this talk presents a declarative programming model +for eventually consistent data stores called Quelea. The model comprises a +contract language, capable of defining fine-grained application-level +consistency properties for replicated data types (and transactions over + objects of these types), and a contract enforcement system to analyze + contracts and automatically generate the appropriate consistency protocol + for the method protected by the contract. By doing so, Quelea enables + programmers to reason compositionally about consistency from the perspective + of high-level application requirements, not low-level implementation + features.

+ +

This is joint work with Gowtham Kaki and K.C. Sivaramakrishnan.

+

Bio

+

Suresh Jagannathan is a Professor of Computer Science at Purdue University + where he has been on leave since September 2013, serving as a program + manager in the Information Innovation Office at DARPA. He has also been a + visiting faculty at Cambridge University, where he spent a sabbatical year + in 2010; and, prior to joining Purdue, was a senior research scientist at + the NEC Research Institute in Princeton, N.J. He received his Ph.D from + MIT.

+ +

His research interests are in programming languages generally, with specific +focus on compilers, functional programming, program verification, and +concurrent and distributed systems. At DARPA, he manages programs on +probabilistic programming and machine learning (PPAML), program synthesis +and repair leveraging predictive analytics over large software corpora +(MUSE), and self-adaptive software through resource-aware analyses, +runtimes, and architectures (BRASS).


+ +
Hop.js: multitier programming in JavaScript +
+Manuel Serrano +INRIA +http://www-sop.inria.fr/members/Manuel.Serrano/ +
+3 November, 2015 10:30AM +WVH 366 +
+

Abstract

+Hop.js is a multitier extension of JavaScript. It allows a single + JavaScript program to describe the client-side and the server-side + components of a Web application. Its runtime environment ensures a + consistent execution of the application on the server and on the + client. This talk will shows its minimal set of JavaScript extensions + that makes Web programming easier. It will present its runtime + environment, with an emphasize on the handling of server-side + parallelism. +

Bio

+Manuel is a researcher at INRIA Sophia Antipolis, he used to work on Scheme.

+ +
Scalloc and Selfie: Fast Memory Allocation and Self-referential Systems Software +
+Christoph Kirsch +University of Salzburg +http://cs.uni-salzburg.at/~ck/ +
+2 November, 2015 11:00AM +WVH 366 +
+

Abstract

+ This talk is about scalloc, a fast, multicore-scalable, low-fragmentation memory allocator and selfie, a 4000-line implementation of a tiny self-compiling C compiler and a tiny self-executing MIPS emulator for teaching systems engineering. Scalloc is a typical example of a very complex, multi-year research effort while selfie is, at least for now, a purely educational, many-year effort in teaching compiler, operating system, and virtual machine design based on a single, highly principled software platform. So far scalloc and selfie only share the passion of their authors and are otherwise two distinct projects. Yet earlier versions of selfie, before they were even identified as such, were instrumental in bringing up the generation of students who did scalloc. +The main ideas behind scalloc are: uniform treatment of small and big objects through so-called virtual spans, efficiently and effectively reclaiming free memory through fast and scalable global data structures, and constant-time (modulo synchronization) allocation and deallocation operations that trade off memory reuse and spatial locality without being subject to false sharing. The main ideas behind selfie are: a compiler written in and for a tiny subset of C called C* which uses the dereferencing * operator of C for memory access but lacks data structures and many other features and a MIPS emulator written in C* that can execute itself. Both are combined and extended by students to do very cool stuff. +

Bio

+ Christoph Kirsch is Professor at the University of Salzburg, Austria. From +1999 to 2004 he worked as Postdoctoral Researcher at UC, Berkeley. He later +returned to Berkeley as Visiting Scholar (2008-2013) and Visiting Professor +(2014) as part of a collaborative research effort in Cyber-Physical +Systems. His most recent research interests are in concurrent data +structures, memory management, and so-called spatial programming. Dr. Kirsch +co-invented embedded programming languages and systems such as Giotto, HTL, +and the Embedded Machine, and more recently co-designed high-performance, +multicore-scalable concurrent data structures and memory management +systems. He co-founded the International Conference on Embedded Software +(EMSOFT) and served as ACM SIGBED chair from 2011 until 2013 and ACM TODAES +associate editor from 2011 until 2014.

+ +
Coordinated Concurrent Programming in Syndicate +
+Tony Garnock-Jones +Northeastern University +http://www.ccs.neu.edu/home/tonyg/ +
+28 October, 2015 12:00PM +WVH 366 +
+

Abstract

+Most programs interact with the world: via graphical +user interfaces, networks, etc. This form of interactivity entails +concurrency, and concurrent program components must coordinate their +computations. This talk will present Syndicate, a design for a coordinated, +concurrent programming language. Each concurrent component in Syndicate is a +functional actor that participates in scoped conversations. The medium of +conversation arranges for message exchanges and coordinates access to common +knowledge. As such, Syndicate occupies a point in design space halfway +between actors and threads. +

Bio

+ Tony is a graduate student working with Matthias.

+ +
Memory corruption: why protection is hard +
+Mathias Payer +Purdue University +https://nebelwelt.net +
+23 October, 2015 1:30PM +WVH 366 +
+

Abstract

+ Memory corruption plagues systems since the dawn of computing. With the +rise of defense techniques like stack cookies, ASLR, and DEP, attacks +have become much more complicated, yet control-flow hijack attacks are +still prevalent. Attacks leverage code reuse attacks, often using some +form of information disclosure. Stronger defense mechanisms have been +proposed but none have seen wide deployment so far (i) due to the time +it takes to deploy a security mechanism, (ii) incompatibility with +specific features, and (iii) most severely due to performance overhead. +In the talk, we evaluate the security benefits and limitations of the +status quo and look into upcoming defense mechanisms (and their attacks). + +Control-Flow Integrity (CFI) and Code-Pointer Integrity (CPI) are two of +the hottest upcoming defense mechanisms. CFI guarantees that the runtime +control flow follows the statically determined control-flow graph. An +attacker may reuse any of the valid transitions at any control flow +transfer. CPI on the other hand is a dynamic property that enforces +memory safety guarantees like bounds checks for code pointers by +separating code pointers from regular data. We will discuss differences +and advantages/disadvantages of both approaches, especially the security +benefits they give under novel attacks like Counterfeit Object-Oriented +Programming (COOP) and Control-Flow Bending (CFB). COOP reuses complete +functions as gadgets and CFB bends the control flow along valid but +unintended paths in the control-flow graph of a program. + +

Bio

+Mathias Payer is a security researcher and an assistant professor in +computer science at Purdue university. His interests are related to +system security, binary exploitation, user-space software-based fault +isolation, binary translation/recompilation, and (application) +virtualization. His research focuses on protecting applications even in +the presence of vulnerabilities, with a focus on memory corruption. +Before joining Purdue in 2014 he spent two years as PostDoc in Dawn +Song's BitBlaze group at UC Berkeley. He graduated from ETH Zurich with +a Dr. sc. ETH in 2012.

+ +
Using LLVM as a backend for R +
+Jan Vitek +Northeastern University +http://janvitek.org +
+2 October, 2015 12:00PM +WVH 366 +
+

Abstract

+I will provide an update on the status of the Reactor project which aims to use LLVM as just-in-time compiler for the R language. I will discuss early challenges such as integration in the environment and garbage collection support. +

Bio

+Jan Vitek is a Professor at Northeastern, he works on programming language design and implementation.

+ +
Completions and Diagnostics in RStudio +
+Kevin Ushey +RStudio +https://kevinushey.github.io/ +
+2 October, 2015 11:00AM +WVH 366 +
+

Abstract

+Kevin will discuss the details behind the implementation of completions + +diagnostics, as well as some future goals re: enabling user extensibility of +both the autocompletion and diagnostics systems. +

Bio

+Kevin is a software engineer at RStudio. He graduated from the University of British Columbia with an MSc in Statistics, with a thesis focusing on the use of non-linear mixed effects models in the analysis of yeast growth curves in gene knockout studies. Shortly thereafter, he worked doing data analysis at St. Paul's hospital in Vancouver as part of the Daley lab, and later at the Fred Hutchinson Cancer Research Center in Seattle as part of the Gottardo lab. At RStudio, Kevin primarily works on the RStudio IDE, but also maintains the R package packrat.

+ +
Some New Developments for the R Engine +
+Luke Tierney +University of Iowa +http://homepage.stat.uiowa.edu/~luke/ +
+2 October, 2015 10:00AM +WVH 366 +
+

Abstract

+ R is a dynamic language for statistical +computing and graphics. In recent years R has become a major framework +for both statistical practice and research. This talk present a very +brief outline of the R language and its evolution and describe some +current efforts on improvements to the core computational engine, +including work on compilation of R code, efforts to take advantage of +multiple processor cores, and modifications to support working with +larger data sets. +

Bio

+Luke Tierney is a Professor Statistics at the University of Iowa and one of the key contributors to the R project.

+ +
Verification Infrastructure for Permission-based Reasoning +
+Alex Summers +ETH Zürich +http://people.inf.ethz.ch/summersa/wiki/ +
+26 June, 2015 1:45PM +WVH 366 +
+

Abstract

+ Modern verification techniques are becoming +ever-more powerful and sophisticated, and building tools to implement +them is a time-consuming and difficult task. Writing a new verifier to +validate each on-paper approach is impractical; for this reason +intermediate verification languages such as Boogie and Why3 have become +popular over the last decade. However, verification approaches geared +around complex program logics (such as separation logic) have typically +been implemented in specialised tools, since the reasoning is hard to +map down to first-order automated reasoning. In practice, this means +that a rich variety of modern techniques have no corresponding tool +support. In this talk, I will present the new Silver intermediate +verification language, which has been designed to facilitate the +lightweight implementation of a variety of modern methodologies for +program verification. In contrast to lower-level verification languages, +Silver provides native support for heap reasoning; modes of reasoning +such as concurrent separation logic, dynamic frames and +rely-guarantee/invariants can be simply encoded. Silver has been +developed as part of the Viper project, which provides two automated +back-end verifiers for Silver programs. Since releasing our software in +September last year, it has been used for (internal and external) +projects to build tools for Java verification, non-blocking concurrency +reasoning, flow-sensitive typing and reasoning about GPU and Linux +kernel code. +

Bio

+Alex Summers obtained his PhD from Imperial College +London in 2009, in the area of type systems and +classical logics. Since then he has worked in a +variety of areas concerning software correctness and +verification, at Imperial College London and ETH +Zurich. His research interests include developing +specification techniques for different (usually +concurrent) programming paradigms, and implementing +these in automatic verification tools. He was +recently awarded the 2015 AITO Dahl-Nygaard Junior +Prize for his work on type systems and the +verification of object-oriented programs.

+ +
Concurrent Trace Sets for Synchronization Synthesis +
+Roopsha Samanta +IST Austria +http://pub.ist.ac.at/~rsamanta/ +
+26 June, 2015 1:30PM +WVH 366 +
+

Abstract

+ In this talk, I will first present a method and +a tool TARA for generating succinct representations of sets of +concurrent traces. In our work, we focus on trace sets that contain all +correct or all incorrect permutations of events from a given trace. We +represent such trace sets as Boolean combinations of happens-before +ordering constraints between events. Our trace set representations can +drive diverse verification, fault localization, repair, and synthesis +techniques for concurrent programs. In the remainder of the talk, I will +focus on the use of our representation for synchronization synthesis. +This work appears in POPL 2015 and CAV 2015, and is joint work with +Pavol Cerny, Ed Clarke, Ashutosh Gupta, Tom Henzinger, Arjun +Radhakrishna, Leonid Ryzhyk and Thorsten Tarrach. +

Bio

+ +

+ +
+ +
+ +

Programming Language Seminar, Junior

+

The junior PL seminar +(mailing list) +(home page): +a regular student-only seminar, suitable for those new to the study of programming languages.

+
+

The PL Seminar, Jr. is a seminar where students new to the area can discuss the study of programming languages. This seminar is not intended to replace the main PL Seminar, but we hope that it can augment that by providing a place where junior students can discuss topics and ask questions at a pace better suited to our level of knowledge. Topics include: +

  • Design and analysis of programming languages: syntax, semantics, pragmatics (ie, how do you use a language feature in real programs?)
    • +
  • Implementation of programming languages
    • +
  • Program development, both large and small
    • +
  • Programming pedagogy
    • +
  • Programming tools and environments
    • +
  • and anything else that catches our interest.
+Topics are not limited to current research, but include older papers, textbook chapters, and surveys.

+
+

Why PL Seminar, Jr.?

+

In many ways, the existing PL Seminar doesn't serve junior students well. For us, it is intended to provide an environment where we can go to listen to current research. While nobody expects us to understand the entire presentation, we would ideally be able to say, "I didn't understand that, but it sounded really neat." By and large, that hasn't happened, for a number of reasons: +

  • For many of us, once our level of comprehension drops below a certain threshold, we tend to get frustrated and tune out. It's very difficult to say that something sounds interesting when you don't even feel like you understand the language.
    • +
  • The discussion environment isn't really helpful to junior students. The questions from the audience are typically research-oriented; often, a great deal of background knowledge is required to follow the ensuing discussions.
    • +
  • To overuse a metaphor, the bandwidth available for discussion tends to be filled by a relatively small number of people; it's often difficult for a junior student to get a word in edgewise.

 +
+

Seminar Culture

+ +

For this seminar to be as useful as possible, it is important we establish the right "seminar culture." This is, of course, something that is hard to describe, and it will shift and evolve. However, there are some guidelines for what we are trying to accomplish: +

  • Attendees are willing to learn and to help others learn. They should not come in order to demonstrate how much smarter they are than everyone else just because they know so much more about a specific topic.
    • +
  • Active listening is encouraged. Ask questions! In particular, while research questions are of course welcome, basic comprehension questions are especially encouraged. If you don't understand something, or if you don't believe a step in a proof, ask!
    • +
  • An important corollary: no question is too stupid to ask. This places an obligation on other members of the seminar: rather than express frustration at the basic nature of a question, look at it as an opportunity to practice your skills in explaining things.
    • +
  • Of course, while active listening is a good thing, we want to avoid having a few people fill the communications channel. So, please be considerate of the other people present and allow them to join the discussions as well.
+

Any attendee may propose a topic; we'll solicit volunteers to present a survey of the area. We expect that most of the presenters will be junior PL students, although outside presentations are welcome.

+

+

© Copyright Programming Research Laboratory 2015-2021 | made by Catchexception s.r.o. | source on GitHub

+top
+ + + + + + + diff --git a/software.html b/software.html new file mode 100644 index 00000000..e1087293 --- /dev/null +++ b/software.html @@ -0,0 +1,66 @@ + + + + + + + + + + +Software - Programming Research Laboratory - Northeastern University + + + + + + + + + + + + +

Software

+ +
We design, implement, and maintain software systems. +

+

The Racket language is a vehicle for most of our research and teaching.

+
+Racket is a full-spectrum programming language. It goes beyond Lisp and Scheme with dialects that support objects, types, laziness, and more. Racket enables programmers to link components written in different dialects, and it empowers programmers to create new, project-specific dialects. Racket's libraries support applications from web servers and databases to GUIs and charts.
+ +

+

The Fiji real-time JVM and Fiji C1 compiler run on on a broad range of HW/OS (ARM and ERC32 to PowerPC and x86/x86_64, from RTEMS to Linux or Darwin), execute Java with deterministic garbage collection, or safe GC-less allocation.

+

+

This project intends to analyze the dynamic behavior of JavaScript programs and its implications on analyses and security.

+ + +

Reactor

+

Tools related to the R programming language including the Purdue implementation of the FastR virtual machine, timeR, testR and benchR.

+ +

+

Larceny is an implementation of the Scheme programming language written by +Will Clinger and his students.

+
+

+

© Copyright Programming Research Laboratory 2015-2021 | made by Catchexception s.r.o. | source on GitHub

+top
+ + + + + + +
diff --git a/teaching.html b/teaching.html new file mode 100644 index 00000000..6092950c --- /dev/null +++ b/teaching.html @@ -0,0 +1,287 @@ + + + + + + + + + + +Teaching - Programming Research Laboratory - Northeastern University + + + + + + + + + + + + +

Teaching

+ +

PRL Faculty & Staff write academic books, teach courses, and write more books.

+ +
+ +
+
H
o
w

t
o

D
e
s
i
g
n +

How to Design Programs
Programs

+

How To Design Programs focuses on the program design process, +distinguishing it from other introductory books. +This approach fosters a variety of skills --- critical reading, +analytical thinking, creative synthesis, and attention to detail. +On the surface we use engaging contexts, our "hello world" program +is an animation, and students have the opportunity to program games, etc.

+

We have spent over twenty years developing the related Program By Design curriculum and have offerings at the middle-school, high-school and university levels. Our material is even used for in-house corporate training.

+

This is the primary textbook for our courses CS 5010 and CS 2500. +www.htdp.org
+http://www.ccs.neu.edu/home/matthias/HtDP2e/
+http://www.programbydesign.org +http://www.bootstrapworld.org


+ +
+

Essentials of Programming Languages is a study of programming languages. Our goal is to provide a deep, working understanding of the essential concepts of programming languages. These essentials have proved to be of enduring importance; they form a basis for understanding future developments in programming languages.

+

Our approach is both theoretical and hands-on. The book provides views of programming languages using widely varying levels of abstraction, maintaining a clear connection between the high-level and low-level views. The text uses interpreters to express the semantics of many essential language elements in a way that is both clear and executable.

+

http://www.eopl3.com

+
E
s
s
e
n
t
i
a
l
s +

Essentials of programming languages +
+of Programming
Languages

+ +
+ +
S
e
m
a
n
t
i
c
s +

Semantics Engineering with PLT Redex +
+Engineering with
PLT Redex

+

Semantics Engineering with PLT Redex is the first comprehensive presentation of reduction semantics in one volume. It also introduces the first reliable and easy-to-use tool set for such forms of semantics.

+

Automatic tool support is critical for rapid prototyping and modeling and this book is addressed to the working semantics engineer. With PLT Redex, semanticists can formulate models as grammars and reduction models on their computers with the ease of paper-and-pencil designs.

+

This text first presents a framework for the formulation of language models, focusing on equational calculi and abstract machines, then introduces PLT Redex, a suite of software tools for expressing these models as PLT Redex models. Finally, experts describe a range of models formulated in Redex.

+

This is the primary textbook for our course CS 7400 - Intensive Principles of Programming Languages.

+

http://redex.racket-lang.org/

+

+ +
+ +

Courses

+
+
  • 7976 Directed Study +
    +     S 2020 Seminar in Technical Writing Mitch Wand +
    +     F 2019 Seminar in Technical Writing link Mitch Wand +
    +     F 2018 Seminar in Technical Writing link Mitch Wand
    • +
  • 7680 Special Topics in Computer Systems +
    +     F 2016 Programming Models for Distributed Computing link Heather Miller
    • +
  • 7580 Special Topics in Software Engineering +
    +     F 2019 Intensive Software, Construction and Engineering link Jan Vitek
    • +
  • 7480 Special Topics in Programming Languages +
    +      F 2024 Formal Security for Cryptography link +Joshua Gancher +
    +      S 2024 Olin Shivers +
    +      S 2021 History of Programming Languages link Matthias Fellsein +
    +      F 2020 Type Systems John Boyland +
    +      S 2019 Gradual Typing and Principled Language Interoperability link Amal Ahmed +
    +      S 2017 History of Programming Languages link Matthias Fellsein +
    +      F 2016 Advanced Program Analysis link Frank Tip +
    +      F 2015 Types, Contracts, and Gradual Typing link Amal Ahmed +
    +      S 2012 Type Systems link Amal Ahmed
    • +
  • 7470 Seminar in Programming Languages +
    +     S 2024 link Arjun Guha
    • +
  • 7400 Intensive Principles of Programming Languages +
    +     S 2020 link Amal Ahmed
    +     S 2017 link Amal Ahmed
    +     F 2016 link Amal Ahmed
    +     S 2016 link Amal Ahmed
    +     S 2015 link Amal Ahmed
    +     S 2014 link Matthias Felleisen
    +     F 2012 link Matthias Felleisen
    +     F 2011 link Matthias Felleisen
    • +
  • 6515 Software Design and Development
    +     S 2013 link Matthias Felleisen
    • +
  • 6240 Parallel Data Processing in MapReduce
    +     F 2017 link Jan Vitek
    +     S 2016 link Jan Vitek
    +     S 2015 link Jan Vitek
    • +
  • 6050 DS Seminar in Data Science
    +     S 2019 Expeditions in Data Science link Jan Vitek
    • +
  • 5963 Relation and Constraint Programming
    +     S 2021 link Jason Hemann
    • +
  • 5500 Foundations of Software Engineering
    +     F 2020 link Mitch Wand, Frank Tip, Jon Bell
    +     F 2019 link Frank Tip
    +     S 2017 link Michael Weintraub, Frank Tip
    • +
  • 5110 DS Introduction to Data Science
    +     S 2017 link Jan Vitek
    • +
  • 5010 Program Design Paradigms
    +     S 2017 link William D. Clinger
    +     F 2016 link Mitch Wand, William D. Clinger, James Miller, Ezra Kilty
    +     S 2016 link William D. Clinger, Kevin Gold
    +     F 2015 link Mitch Wand, William D. Clinger
    +     S 2015 link Stephen Chang
    +     F 2014 link Mitch Wand, Stephen Chang, Jan Vitek
    +     S 2014 link Ryan Culpepper
    +     F 2013 link Ryan Culpepper
+
  • 4973/7580 Advanced Software Engineering
    +     S 2023 link Jon Bell
    • +
  • 4910 Type Systems
    +     F 2020 John Boyland
    • +
  • 4620 Building Extensible Systems
    +     S 2018 Hack Your Own Language link Matthias Felleisen, Stephen Chang
    +     S 2015 link Matthias Felleisen
    • +
  • 4500 Software Development
    +     F 2023 link Matthias Felleisen, Ben Lerner
    +     F 2022 link Matthias Felleisen, Ben Lerner
    +     F 2020 link Matthias Felleisen, Jason Hemann
    +     S 2020 link Jan Vitek, Jason Hemann, Ferdinand Vesely
    +     F 2019 link Matthias Felleisen
    +     F 2018 link Matthias Felleisen, Jason Hemann
    +     S 2016 link Matthias Felleisen
    • +
  • 4410/6410 Compilers
    +     S 2024 link Ben Lerner
    +     S 2023 link Ben Lerner
    +     F 2022 link Olin Shivers
    +     S 2021 link Ben Lerner
    +     S 2020 link Ben Lerner
    +     F 2019 link Olin Shivers
    +     S 2019 link Ben Lerner
    +     F 2018 link Olin Shivers
    +     S 2017 link Ben Lerner
    +     F 2013 link Amal Ahmed
    +     S 2012 link Olin Shivers
    • +
  • 4100 Artificial Intelligence
    +     F 2022 Steven Holtzen
    • +
  • 4400 Programming Languages
    +     S 2024 Steven Holtzen
    +     S 2021 link Jason Hemann
    +     S 2020 link Matthias Felleisen
    +     F 2018 link Jason Hemann
    • +
  • 4530 Fundamentals of Software Engineering
    +     S 2024 link Mitch Wand, Jon Bell
    +     F 2023 link Frank Tip
    +     F 2023 link Mitch Wand
    +     S 2023 link Mitch Wand, Jan Vitek
    +     F 2022 link Mitch Wand, Jon Bell
    +     S 2021 link Jon Bell, John Boyland, Mitch Wand
    +     F 2020 link Frank Tip, Jon Bell
    • +
  • 3800 Theory of Computation
    +     F 2015 link William D. Clinger
    • +
  • 3520 Programming in C++
    +     S 2023 James Perretta
    • +
  • 3500 Object Oriented Design
    +     F 2023 link Ben Lerner
    +     F 2022 Ben Lerner
    +     F 2019 link Ben Lerner, Clark Freifeld, Alex Grob
    +     F 2018 link Ben Lerner, Amit Shesh, Clark Freifeld
    +     F 2016 link Ben Lerner, Amit Shesh
    +     F 2015 link Ben Lerner
    • +
  • 2800 Logic and Computation
    +     S 2024 link Daniel Patterson
    +     F 2023 Olin Shivers
    +     S 2023 link Daniel Patterson
    +     S 2021 link Jason Hemann
    +     F 2020 link Jason Hemann
    +     S 2020 link Pete Manolios, Olin Shivers
    • +
  • 2510 Fundamentals of Computer Science 2
    +     S 2024 link Ben Lerner
    +     S 2024 accelerated link Ben Lerner
    +     S 2023 link Ben Lerner
    +     S 2023 accelerated link Ben Lerner
    +     S 2021 accelerated link Ben Lerner
    +     S 2020 accelerated link Ben Lerner
    +     S 2019 link Leena Razzaq, Jason Hemann, Matthew Singer
    +     S 2019 accelerated link Ben Lerner, Matthew Singer
    +     S 2018 link Ben Lerner, Nada Naji, Clark Freifeld, Jan Vitek, Becca MacKenzie
    +     S 2018 accelerated link Ben Lerner, Becca MacKenzie
    +     S 2017 link Ben Lerner, Nada Naji, Leena Razzaq, Clark Freifeld, Becca MacKenzie
    +     S 2016 link Ben Lerner, Nada Naji, Leena Razzaq, Magy Seif El-Nasr
    +     S 2015 link Ben Lerner, Leena Razzaq
    +     F 2014 link Ben Lerner, Leena Razzaq
    +     S 2014 honors link Ben Lerner
    • +
  • 2500 Fundamentals of Computer Science
    +     F 2023 link Daniel Patterson, Arjun Guha
    +     S 2023 link Olin Shivers
    +     F 2022 accelerated link Arjun Guha
    +     F 2022 link Nate Derbinsky, John Park, Daniel Patterson, Leena Razzaq, Olin Shivers
    +     F 2020 accelerated link Amal Ahmed, Ben Lerner
    +     F 2020 link Amal Ahmed, Ben Lerner, Arjun Guha, John Park, Ferdinand Vesely
    +     F 2019 accelerated link Amal Ahmed
    +     S 2019 link Olin Shivers
    +     F 2018 accelerated link Amal Ahmed
    +     F 2017 link Ben Lerner, Alan Mislove, Christo Wilson, Nada Naji, Byron Wallace
    +     F 2017 accelerated link Matthias Felleisen
    +     S 2017 link Olin Shivers, Nat Tuck
    +     F 2016 link Matthias Felleisen, Amal Ahmed, Ben Lerner
    +     F 2016 accelerated link Olin Shivers
    +     F 2015 link Olin Shivers
    +     F 2015 honors link Ben Lerner
    +     F 2014 link Olin Shivers, Amal Ahmed, Ben Lerner
    +     S 2014 link Amal Ahmed
    +     F 2013 link Matthias Felleisen, Amal Ahmed
    +     F 2011 link Amal Ahmed
    • +
  • 1101 Computer Science and its Applications
    +     S 2024 link Karl Lieberherr
    +     F 2023 link Karl Lieberherr
    +     S 2023 link Karl Lieberherr
    +     F 2022 link Karl Lieberherr
+ +
+ +
+

Racket is a descendant of Lisp, a programming language renowned for its elegance, power, and challenging learning curve. But while Racket retains the functional goodness of Lisp, it was designed with beginning programmers in mind and Realm of Racket is an introduction to the Racket language.

+

Our approach teaches programming by creating increasingly complex games. The journey begins with the Guess My Number game and coverage of some basic Racket etiquette. Next, readers dig into syntax and semantics, lists, structures, and conditionals, and learn to work with recursion and the GUI in building the Robot Snake game. Then it's on to lambda and mutant structs (and an Orc Battle), and fancy loops and the Dice of Doom. Finally, readers explore laziness, AI, distributed games, and the Hungry Henry game.

+

http://realmofracket.com/realmofracket.com

+
+

Realm of Racket +
+Realm of Racket

+ +
+ +

The Littles Series +
+The Littles Series

+

The Littles Series open new doors of thought to everyone who wants to find out what computing is really about.

+

The original, Little LISPer, unfolds some of the most beautiful concepts in mathematics, computer science, and logic. The follow-on books further the notion that 'thinking about computing is one of the most exciting things the human mind can do.' This sets both The Little Schemer and its companion volume, The Seasoned Schemer, apart from other books on LISP.

+

The Little MLer introduces one of the most important members of the family of programming languages. ML has emerged as a natural language for software engineering courses because it provides a sophisticated and expressive module system and is the language of choice for some NU CCIS courses.

+

Design patterns, which moved object-oriented programming to a new level, provide programmers with a language to communicate with others about their designs. As a result, programs become more readable, more reusable, and more easily extensible. A Little Java, A Few Patterns, use a small subset of Java to introduce pattern-directed program design.

+

+

© Copyright Programming Research Laboratory 2015-2021 | made by Catchexception s.r.o. | source on GitHub

+top
+ + + + + + +